Senior Applied Scientist - AI Platform

At Datadog, a Most Loved Workplace® certified employer in the Technology space, Observe, Secure, and Optimize Modern Cloud Infrastructure at Scale.

<p>AI Platform builds the foundations of Datadog's AI efforts. The org is 70+ people organised in three pillars: training and serving (GPU clusters, distributed training, low-level infrastructure), agents (agent harnesses, memory systems, the internal AI gateway that routes every LLM request at Datadog), and evaluation and experimentation. This role sits in the evaluation and experimentation pillar, which owns Datadog's shared annotation and evaluation infrastructure — including the evaluation scenario store and the telemetry archival systems used across the Bits org. Together they let an agent travel back in time and query what Datadog looked like at the exact moment an incident happened, so scenarios can be replayed and agent performance tracked over time.</p>

<p>Specifically, you'll be the first applied scientist on GenSim (Generative Simulations), the team that builds the environments Datadog's agents learn in. GenSim doesn't replay sampled telemetry — it stands up real, fully instrumented applications that talk to Datadog, drives them with representative traffic, injects controlled failures, and records what happens. Because GenSim injected the failure, it knows the ground truth. That corpus — hundreds of postmortem-derived scenarios and thousands of runnable applications — is today the primary source of post-training data for Datadog's own SRE model, and the substrate that Bits AI SRE and our other agents are trained and evaluated against.</p>

<p>The team has no applied science support today and is learning post-training data methodology on the fly. That's the gap this role fills, and the open questions are the interesting part. How do you tell whether a generated environment is actually representative of the messy, incomplete telemetry real customers run — rather than a suspiciously clean one where every monitor exists and every service emits complete logs? How do you make injected problems genuinely h

Want to learn more about what it's like to work at Datadog? View our full profile.