Senior Machine Learning Engineer - Search and Recommendations

Overview

About the Role:

This is a general posting for multiple Sr. Machine Learning roles open across our 4-sided marketplace. You’ll get the chance to learn about the problems the different ML teams solve as you go through the process. Towards the end of your process, we’ll do a team-matching exercise to determine which of the open roles/teams you’ll join. You can find a blurb on each team at the bottom of this page. 

About the Team:

The Search & Recommendations ML team sits at the forefront of Instacart’s push to unify search, discovery, merchandising, and recommendations under a single, reasoning-rich AI platform. Working shoulder-to-shoulder with world-class engineers, data scientists, and product managers, we build the LLM backbone that will power every pixel of the shopping journey. As part of our team, you'll tackle one of the most critical aspects of the business—helping customers connect with exactly the right products through AI.  

What we’re building:

• Models for Search and Recommendations – We’re consolidating over 80 task-specific models into a small set of large, general-purpose LLMs and multi-task models that reason holistically across user intent understanding, ranking, ads, and merchandising. 
• AI-driven merchandising – Real-time orchestration of carousels and bundles that brings thematic cohesion (“Taco Tuesday”, “Back-to-School Lunches”) to storefronts while balancing relevance and revenue.
• Agentic cart starters & basket building – LLM agents that analyze purchase history, preferences, and promos to pre-populate carts with highly probable items, boosting basket size and reducing friction.

Our commitment to AI innovation is reflected in our recent publications and research contributions to the field (Recent publications 1, 2, 3, 4, 5, 6). 

About the Job:

• Architect and scale foundational LLM systems that unify query understanding, personalization, ranking, ads, and merchandising—replacing dozens of siloed models with a single, adaptive backbone.
• Develop agentic workflows that proactively support shoppers by generating personalized cart starters, thematic bundles, and dynamic storefronts—driven by LLMs reasoning over historical behavior, preferences, and context.
• Fine-tune large language models using SFT, DPO, and GRPO on rich behavioral feedback signals to align system outputs with evolving customer needs and business objectives.
• Design intelligent, end-to-end discovery pipelines that handle everything from long-tail semantic retrieval to real-time multi-objective ranking and merchandising optimization.
• Collaborate cross-functionally with product, design, infra, and ads teams to translate high-level discovery goals into scalable LLM-powered systems that drive measurable impact.
• Coach and mentor a team of ML engineers, fostering their technical and professional growth

About You:

Minimum Qualifications:

• Have 5+ years of industry experience using machine learning to solve real-world problems with large datasets with 3+ years in a technical leadership role
• Proven track record designing and deploying sophisticated ML/AI systems in production environments that drive measurable business impact through improved recommendations, search relevance, and user engagement metrics.
• Have strong programming skills in Python and fluency in data manipulation (SQL, Pandas) and Machine Learning (scikit-learn, XGBoost, Keras/Tensorflow) tools
• Have strong analytical skills and problem-solving ability
• Are a strong communicator who can collaborate with diverse stakeholders across all levels

Preferred Qualifications:

• Extensive expertise with modern deep learning frameworks (PyTorch, TensorFlow, JAX) and advanced LLM architectures including transformer models, attention mechanisms, and multimodal AI systems.
• Demonstrated experience implementing and fine-tuning large language models, including prompt engineering, embedding techniques, and efficient inference optimization for production environments.
• Strong foundation in AI fundamentals including neural network architectures, generative models, and foundation model adaptation methodologies like PEFT, LoRA, and RLHF.
• Experience optimizing AI model performance across the full stack, from model architecture and training workflows to distributed inference and serving infrastructure.
• Self-motivated innovator with a strong sense of ownership who can navigate the rapidly evolving AI landscape, evaluate emerging techniques, and implement novel approaches to solve complex business challenges.
• Passion for applying cutting-edge AI research to real-world applications and a keen understanding of the practical considerations in developing responsible, efficient AI systems at scale.