Principal Software Engineer

The HPC/AI (High Performance Computing and Artificial Intelligence) team is building the infrastructure that powers large-scale AI training and inference. As supercomputers grow across first-party (1P) and third-party (3P) GPU capacity, we need a consistent, trusted view of their health and operational state. 

We are seeking an AI Supercomputer Observability Architect to define and deliver the end-to-end observability platform for this fleet. You will drive near-real-time telemetry from GPUs, hosts, networks, storage, and other critical components; connect those signals to resource identity and topology; and turn an ocean of data into actionable, AI-assisted intelligence. The goal is to help customers and operators identify impact, find likely causes, enable datacenter technicians to diagnose and repair issues quickly, verify each repair with immediate telemetry feedback, restore service faster, and use GPU capacity more effectively. 

This is a horizontal architecture role. You will align infrastructure, platform, operations, and partner teams on common contracts and a converged 1P/3P experience, while accounting for differences in data availability, quality, and access. You will move the work from architecture through production validation, with a focus on scalability, reliability, security, and measurable customer outcomes. 

Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees, we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond. 

•  Lead the architecture for near-real-time telemetry ingestion, processing, storage, and query across GPUs, hosts, InfiniBand, Ethernet, NVLink, storage, power, cooling, and related supercomputer components. 

•  Define shared signal contracts and measurable standards for freshness, completeness, correctness, availability, retention, access, and cost; validate the end-to-end pipeline under production-scale conditions. 

•  Connect telemetry to a common resource identity, topology, and health model so users can navigate from fleet and cluster health to racks, nodes, GPUs, switches, and links, and understand dependencies and blast radius. 

•  Drive convergence of 1P and 3P GPU capacity telemetry into a consistent operational model and experience, making provider-specific gaps and differences in signal quality explicit. 

•  Build the foundations for AI-assisted anomaly detection, cross-signal correlation, likely-cause analysis, impact assessment, and actionable recommendations; establish evaluation and feedback loops so insights are trustworthy. 

•  Partner with customers, datacenter technicians, service owners, and engineering DRIs to define persona-appropriate workflows and secure, role-based views of the same underlying state. 

•  Build practical tools and dashboards for datacenter technicians that surface near-real-time component health, likely fault location, repair context, and the next diagnostic step, so technicians can act quickly at the rack or node. 

•  Create a fast repair-validation loop: show technicians whether a repair changed the relevant signals, whether the component and affected capacity recovered, and what remains unhealthy so they can iterate until the problem is resolved. 

•  Lead technical design reviews, identify cross-team dependencies, establish reusable interfaces and patterns, and guide teams from proof of concept through incremental production rollout. 

•  Define success measures such as signal freshness, coverage, time to diagnose, time to repair, repair-verification latency, repeat repair attempts, recovery time, and the ability to connect infrastructure events to affected GPU capacity and customer impact. 

•  Mentor engineers and influence teams across organizational boundaries; contribute to operational playbooks and continuously improve the platform based on production learning.