Interested in this AI/ML Engineer role at Pantheon Data?
Apply Now →Skills & Technologies
About This Role
Company Overview
Pantheon Data (a Kenific Holding company) is a private, small business based in the Washington, DC, area. Pantheon Data was founded in 2011, initially providing acquisition and supply chain management services to the US Coast Guard. Our service offerings have grown in the past ten years, including infrastructure resiliency, contact center operations, information technology, software engineering, program management, strategic communications, engineering, and cybersecurity. We have also grown our customer base to include commercial clients. The company has used this experience to expand our service offerings to other agencies within the Department of Homeland Security (DHS), the Department of Defense (DoD), and other Federal Civilian Agencies.
Position Overview
Pantheon Data is seeking a Senior AI Engineer (Agentic Systems) to design, build, and operate the agentic framework at the core of our AI platform in AWS GovCloud, serving workloads that process CUI, PII, and PHI. You will own the framework that routes requests to the right agents, composes new agents at runtime from a registry of capabilities, and coordinates their tools, state, and handoffs, so that these systems behave predictably, securely, and auditably under FedRAMP High and DoD IL4/IL5.
This role owns the agent runtime: orchestration, routing, the capability registry, and the contracts that capabilities plug into. Capabilities such as retrieval, document extraction, and application integrations are built by machine learning and software engineers and exposed through the registry; you define how they are described, discovered, composed, and governed. You will partner closely with the Data Engineer, who owns the pipelines and data stores agents draw on; the Cloud Security Engineer, who owns cloud security controls and ATO evidence; and the DevSecOps and Cloud Operations Engineers, who own delivery and infrastructure. This is a systems engineering role, not a prompt-only or prototype role: successful candidates can walk through production agentic systems they have personally built, including the orchestration design, how routing decisions were made and evaluated, how failures were contained, and what they would change.
Responsibilities
- Agent Orchestration: Design and build the orchestration layer that plans, dispatches, and coordinates work across agents, including multi-step and multi-agent workflows, handoffs, parallel execution, cancellation, and human-in-the-loop checkpoints.
- Routing: Build the routing that maps requests and intermediate steps to the right agents, tools, and models, combining deterministic rules, classifiers, and LLM-based routing where each is appropriate; measure routing accuracy and cost, and make routing decisions observable and explainable.
- Capability Registry and Dynamic Agents: Own the capability registry, including schemas for describing capabilities, versioning, discovery, and access policy, and the mechanisms that assemble agents at runtime from registered capabilities, tools, and instructions.
- Tool and Integration Contracts: Define the interfaces capabilities plug into, including tool definitions, Model Context Protocol (MCP) servers, input and output schemas, and error semantics, and help capability owners onboard to them.
- State, Memory, and Reliability: Design agent state, session memory, and context management; build retries, timeouts, idempotency, loop and budget limits, and graceful degradation so agent failures are contained and recoverable.
- Evaluation and Observability: Build evaluation harnesses for agent behavior, including routing accuracy, task completion, tool-call correctness, and regression across model and prompt versions, and trace every agent run end to end for debugging, cost, and latency.
- Agent Security and Auditability: Enforce least-privilege tool access per agent and per request, contain prompt injection across tool and agent boundaries, and record a complete audit trail of agent decisions and actions; work with the Cloud Security Engineer to map these controls to NIST SP 800-53 and support ATO.
- Technical Leadership: Set the architecture and standards for agentic development across the team, review designs and code, and mentor engineers building capabilities on the framework.
Required Skills and Experience
- Bachelor's degree in Computer Science, Engineering, or a related technical field from an accredited university, or equivalent professional experience.
- 8+ years of professional software engineering experience, including 2+ years building production LLM-backed systems.
- Demonstrated experience designing and shipping a production agentic system: multi-step or multi-agent orchestration with tool use, beyond single-prompt chat applications or demos.
- Hands-on experience with LLM tool calling, structured outputs, and at least one agent framework or SDK, such as LangGraph, Strands Agents, the Claude Agent SDK, Semantic Kernel, or a custom framework you built.
- Strong software architecture skills: plugin or registry-based designs, interface and schema design, versioning, and extensibility.
- Experience with distributed and asynchronous systems: queues, event-driven workflows, concurrency, retries, idempotency, and failure handling.
- Experience evaluating nondeterministic systems, including building test sets, automated evaluations, and regression checks for LLM behavior.
- Strong Python engineering skills: readable, maintainable, tested code; debugging; packaging; and API and service design.
- Working knowledge of AI security risks such as prompt injection, excessive tool permissions, and data exfiltration, and of controls that mitigate them.
- Experience with AWS services relevant to this work, such as Amazon Bedrock, Lambda, Step Functions, ECS or EKS, SQS, DynamoDB, and CloudWatch.
- Experience with Git-based workflows, code review, CI/CD, and team-based software delivery.
- Clear written and verbal communication, including explaining agent behavior, tradeoffs, and failure modes to technical and non-technical stakeholders.
- Ability to work effectively remotely in cross-functional teams.
- Ability to meet deadlines and produce quality work.
- Proficient in Microsoft Suite software including Outlook, Word, Excel, SharePoint, and PowerPoint.
Preferred Skills and Experience
- Experience building or operating agentic systems in AWS GovCloud or other environments supporting FedRAMP High or DoD IL4/IL5 workloads.
- Experience implementing or integrating MCP servers and clients, or agent-to-agent protocols, in production.
- Experience with Amazon Bedrock beyond basic inference: Agents, Guardrails, Knowledge Bases, AgentCore, provisioned throughput, and cross-model routing.
- Experience building model routing or cascading strategies that balance quality, latency, and cost across multiple models.
- Experience with agent observability and tracing tools such as OpenTelemetry, Langfuse, LangSmith, or Arize Phoenix.
- Experience integrating retrieval, document processing, or other ML capabilities as tools within an agentic framework.
- Familiarity with federal compliance frameworks and their engineering implications: NIST SP 800-53, CUI handling, audit logging, and least-privilege access in regulated environments.
- Experience with infrastructure as code (Terraform), containerized deployment, and GitLab CI or GitHub Actions in regulated environments.
- Advanced degree in a technical discipline, or demonstrated contribution to team technical direction and mentorship of other engineers.
Clearance Requirements
Applicants selected will be subject to a security investigation and may need to meet eligibility requirements. Candidates must have the ability to obtain and maintain a Secret clearance; an active Secret clearance is preferred.
Work Location: Reston, VA - Remote
- Our company prioritizes the benefits of flexibility and collaboration, whether that happens in person or remotely.
- If the position is remote or hybrid, you may periodically work from a Pantheon Data office location or client site.
- If this position is assigned to a Pantheon Data office location or client site, you'll work with colleagues and clients in person, as needed for specific client requirements.
Interview Requirement: Candidates who are local to the area should be prepared to participate in an in-person interview as part of the selection process. Candidates outside the local area may be considered for a virtual interview.
Compensation
The salary range for this position is $120,000 - $220,000. This is not, however, a guarantee of compensation or salary. Rather, salary will be set based on experience, geographic location and possibly contractual requirements and could fall outside of this range.
Benefits Overview
We are always looking for good people! Pantheon Data is committed to providing its employees with competitive salaries and benefits in order to increase employee satisfaction and productivity.In addition to our benefits, we also offer SmartBenefits through the Washington Metro Area Transportation Authority, where you specify an amount of your pre-tax wages be paid directly to your SmarTrip account. In some cases, tuition assistance may be available for continuing education expenses and certifications related to their position. Additional details may be found at https://pantheon-data.com/careers/
Pantheon Data Important Information
All qualified applicants will be considered for employment without regard to disability, status as a protected veteran, or any other status protected by applicable federal, state, local, or international law.
As part of the application process, you are expected to be on camera during interviews and assessments. We reserve the right to take your picture to verify your identity and prevent fraud.
If you require reasonable accommodation in completing this application, interviewing, completing any pre-employment testing, or otherwise participating in the employee selection process, please direct your inquiries to our Talent Team at Recruiting@pantheon-data.com or by phone (571) 363-4020.
This company uses E-Verify to confirm each employee's work authorization. For more information, click here E-Verify Participation Poster
Salary Context
This $120K-$220K range is below the median for AI/ML Engineer roles in our dataset (median: $180K across 1953 roles with salary data).
View full AI/ML Engineer salary data →Role Details
About This Role
AI/ML Engineers build and deploy machine learning models in production. They work across the full ML lifecycle: data pipelines, model training, evaluation, and serving infrastructure. The role has evolved significantly over the past two years. Where ML Engineers once spent most of their time on model architecture, the job now tilts heavily toward inference optimization, cost management, and integrating LLM capabilities into existing systems. Companies want engineers who can ship production systems, and the experimenter-only role is fading fast.
Day-to-day, you're writing training pipelines, debugging data quality issues, setting up evaluation frameworks, and figuring out why your model performs differently in staging than it did on your dev set. The best ML engineers are obsessive about reproducibility and measurement. They instrument everything. They know that a model is only as good as the data feeding it and the infrastructure serving it.
Across the 10,419 AI roles we're tracking, AI/ML Engineer positions make up 75% of the market. At Pantheon Data, this role fits into their broader AI and engineering organization.
Demand for AI/ML Engineers has been strong and consistent. Unlike some AI roles that spike with hype cycles, ML engineering is a foundational need. Every company deploying AI models needs people who can keep them running, and the gap between research prototypes and production systems keeps growing.
What the Work Looks Like
A typical week might include: debugging a data pipeline that's silently dropping 3% of training examples, running A/B tests on a new model version, writing documentation for a feature flag system that lets you roll back model deployments, and reviewing a junior engineer's PR for a new evaluation metric. Meetings tend to be cross-functional since ML touches product, engineering, and data teams.
Demand for AI/ML Engineers has been strong and consistent. Unlike some AI roles that spike with hype cycles, ML engineering is a foundational need. Every company deploying AI models needs people who can keep them running, and the gap between research prototypes and production systems keeps growing.
Skills Required
Python and PyTorch dominate the requirements. Most roles expect experience with cloud platforms (AWS, GCP, or Azure) and familiarity with ML frameworks like TensorFlow or JAX. RAG (Retrieval-Augmented Generation) has become a top-3 skill requirement as companies integrate LLMs into their products. Docker and Kubernetes show up in about a third of postings, reflecting the production focus of the role.
Beyond the core stack, employers increasingly want experience with experiment tracking tools (MLflow, Weights & Biases), feature stores, and vector databases. Fine-tuning experience is valuable but less common than you'd think from reading Twitter. Most production LLM work is RAG and prompt engineering, not fine-tuning. If you have both, you're in a strong position.
Companies that are serious about AI/ML hiring tend to post specific infrastructure details in the job description: the frameworks they use, their model serving stack, their data pipeline tools. Vague postings that just say 'ML experience required' without specifics are often companies that haven't figured out what they need yet.
Compensation Benchmarks
AI/ML Engineer roles pay a median of $213,800 based on 16,346 positions with disclosed compensation. Senior-level AI roles across all categories have a median of $230,000. This role's midpoint ($170K) sits 20% below the category median. Disclosed range: $120K to $220K.
Across all AI roles, the market median is $214,834. Top-quartile compensation starts at $266,000. The 90th percentile reaches $321,730. For comparison, the highest-paying categories include AI Safety ($274,500) and Research Engineer ($252,000). By seniority level: Entry: $114,400; Mid: $193,200; Senior: $230,000; Director: $272,100; VP: $235,000.
Pantheon Data AI Hiring
Pantheon Data has 2 open AI roles right now. They're hiring across AI/ML Engineer, Data Engineer. Based in Reston, VA, US. Compensation range: $190K - $220K.
Location Context
Across all AI roles, 6% (580 positions) offer remote work, while 9,776 require on-site attendance. Top AI hiring metros: New York (3,785 roles, $222,480 median); San Francisco (3,262 roles, $261,300 median); Los Angeles (1,896 roles, $210,000 median).
Career Path
Common paths into AI/ML Engineer roles include Data Scientist, Software Engineer, Research Engineer.
From here, career progression typically leads toward ML Architect, AI Engineering Manager, Principal ML Engineer.
The fastest path into ML engineering is through software engineering with a self-directed ML education. A CS degree helps, but production engineering skills matter more than academic credentials. Build something that works, deploy it, and measure it. That portfolio project is worth more than a Coursera certificate. For career growth, the fork comes around the senior level: go deep on technical complexity (staff/principal track) or move into managing ML teams.
What to Expect in Interviews
Expect system design questions around ML pipelines: how you'd build a training pipeline for a specific use case, handle data drift, or design A/B testing infrastructure for model deployments. Coding rounds typically involve Python, with emphasis on data manipulation (pandas, numpy) and algorithm implementation. Take-home assignments often ask you to build an end-to-end ML pipeline from raw data to deployed model.
When evaluating opportunities: Companies that are serious about AI/ML hiring tend to post specific infrastructure details in the job description: the frameworks they use, their model serving stack, their data pipeline tools. Vague postings that just say 'ML experience required' without specifics are often companies that haven't figured out what they need yet.
AI Hiring Overview
The AI job market has 10,419 open positions tracked in our dataset. By seniority: 117 entry-level, 4,975 mid-level, 3,840 senior, and 1,487 leadership roles (Director, VP, C-Level). Remote roles make up 6% of the market (580 positions). The remaining 9,776 roles require on-site or hybrid attendance.
The market median for AI roles is $214,834. Top-quartile compensation starts at $266,000. The 90th percentile reaches $321,730. Highest-paying categories: AI Safety ($274,500 median, 90 roles); Research Engineer ($252,000 median, 580 roles); AI Engineering Manager ($250,811 median, 72 roles).
Demand for AI/ML Engineers has been strong and consistent. Unlike some AI roles that spike with hype cycles, ML engineering is a foundational need. Every company deploying AI models needs people who can keep them running, and the gap between research prototypes and production systems keeps growing.
The AI Job Market Today
The AI job market spans 10,419 open positions across 16 role categories. The largest categories by volume: AI/ML Engineer (7,849), AI Product Manager (516), AI Software Engineer (513). These three account for the majority of open positions, though smaller categories often have higher per-role compensation because of specialized skill requirements.
The seniority mix tells a story about where AI teams are in their maturity. Entry-level roles (117) are outnumbered by mid-level (4,975) and senior (3,840) positions, reflecting that most companies are past the 'build a team from scratch' phase and need experienced engineers who can ship production systems. Leadership roles (Director, VP, C-Level) total 1,487 positions, representing the bottleneck between technical execution and organizational strategy.
Remote work availability sits at 6% of all AI roles (580 positions), with 9,776 requiring on-site or hybrid attendance. The remote share has stabilized after the post-pandemic correction. Senior and specialized roles (Research Scientist, ML Architect) are more likely to be remote-eligible than entry-level positions, partly because experienced hires have more negotiating power and partly because these roles require less hands-on mentorship.
AI compensation is structured in clear tiers. The market median sits at $214,834. Top-quartile roles start at $266,000, and the 90th percentile reaches $321,730. These figures include base salary with disclosed compensation. Total compensation (including equity, bonuses, and sign-on) runs 20-40% higher at companies that offer those components.
Category matters for compensation. AI Safety roles lead at $274,500 median, while Prompt Engineer roles sit at $140,000. The spread between highest and lowest-paying categories reflects the premium on specialized technical skills versus broader analytical roles.
The most in-demand skills across all AI postings: Python (2,350 postings), Aws (1,173 postings), Azure (1,030 postings), Rag (986 postings), Gcp (714 postings), Pytorch (649 postings), Kubernetes (583 postings), Claude (581 postings). Python dominates, appearing in the vast majority of role descriptions regardless of category. Cloud platform experience (AWS, GCP, Azure) is the second most common requirement. The newer entrants to the top skills list (RAG, vector databases, LLM APIs) reflect the shift from traditional ML toward generative AI applications.
Frequently Asked Questions
Get Weekly AI Career Intelligence
Salary data, skills demand, and market signals from 16,000+ AI job postings. Every Monday.