Interested in this AI Agent Developer role at Chatham Financial?
Apply Now →Skills & Technologies
About This Role
Job Description:
Overview:
We don’t simply hire employees. We invest in them. When you work at Chatham, we empower you — offering professional development opportunities to help you grow in your career, no matter if you've been here for five months or 15 years. Chatham has worked hard to create a distinct work environment that values people, teamwork, integrity, and client service. You will have immediate opportunities to partner with talented subject matter experts, work on complex projects, and contribute to the value Chatham delivers every day.
We are seeking an AI Agent Developer to design, build, and deploy AI agents that transform how teams at Chatham work. This is a hands-on engineering role for someone who wants to apply modern AI to real business problems and see the impact firsthand. You will report to one of our Software Engineering Directors to maintain alignment with our organizational AI objectives.
In each role, you will embed with a partner team as a forward-deployed engineer. You will review past work, shadow team members on live tasks, and map where time is spent and why. From there, you will identify and prioritize the highest-value opportunities, then build, evaluate, and release agents that triage requests, research and draft answers, investigate issues across systems and data, and prepare work for review. The team’s experts stay in the loop throughout, and people review agent output before it reaches a client or a production system.
Once a team’s agents are running reliably, you will hand them over to the people who own the work and bring the same approach to the next team. Over time, you will partner with teams across the firm, from engineering and operations to client service and Marketing, wherever repetitive work can be improved through agentic means. You will work closely with Chatham’s AI evaluation and platform teams and help set the standards that other teams adopt.
This is a new role at Chatham and an opportunity to help define how the firm builds and scales agentic capabilities. The ideal candidate is a pragmatic engineer who enjoys learning new domains from the people who do the work, collaborates easily with technical and non-technical colleagues, and measures success by the problems solved for users.
In this role you will:
Discovery & Partnership
- Embed in a partner team and learn how it works. Review past work, shadow live tasks, and map the steps that the team repeats.
- Find and rank the tasks where agents can save the most time or improve quality and agree on priorities with the team’s leaders.
- Work closely with the team’s experts. They supply the domain knowledge and test cases, and they review agent output.
Agent Development
- Design, build, and deploy cloud-native AI agents that do repetitive work from documented processes, with a person in the loop where judgment is necessary.
- Build agents that triage and route incoming requests, research questions and draft answers, and investigate issues across systems, logs, and data.
- Build agents that prepare work for people to review, such as draft responses, reports, and code changes as merge requests.
- Turn each team’s playbooks and procedures into agent instructions and tools and keep them current as processes change.
Evaluation & Quality
- Create evaluation sets from real past work. Measure each agent’s accuracy before release and whenever models or techniques change and monitor the results in production.
- Partner with our AI evaluation team, and adopt its tools, techniques, and processes as they mature.
- Follow Chatham’s standards for security, data handling, and the responsible use of AI.
Reuse & Handover
- Document each agent’s purpose, tools, and limits so that other teams can adopt and extend it.
- Build reusable tools and patterns that make each new team’s agents faster to deliver.
Hand each team’s agents over to the people who own the work, then move to the next team.
*
Your Impact:
For each partner team
- Agents do a growing share of the team’s routine work correctly, and people review the output.
- The team spends less time on repetitive tasks and more time on work that needs judgment.
- Requests that wait in a queue today get faster and more consistent answers.
- Each agent meets its accuracy targets before release and keeps them in production.
Across Chatham
- Each agent has documentation and tests, and other teams reuse its tools and patterns.
- Each engagement ends with a clean handover to the people who own the work.
A ranked list of agent opportunities is ready for the next team before you move to it.
*
Contributors to your success:
- 3+ years of professional software engineering experience.
- Strong proficiency in Python or C#, with experience in APIs, services, or automation.
- Hands-on experience with applications built on large language models (LLMs), including prompt and context design, tool use, and agent workflows.
- Experience with evaluation of AI output: test sets, accuracy measures, and regression checks for systems that do not always return the same answer.
- Solid SQL skills, and comfort with logs and telemetry to find the root cause of a problem.
- Experience with Git, code review, and CI/CD.
- Curiosity and patience to learn complex business domains from the people who do the work.
- Ability to turn a business process into clear steps that an agent can follow, and to judge where a person must stay in the loop.
- Clear written and verbal communication with technical and non-technical colleagues.
- Experience with tools and integrations built on the Model Context Protocol (MCP).
- Experience with platforms or frameworks for AI agents, such as Microsoft Foundry or the Claude Agent SDK.
- Daily use of AI coding tools such as Claude Code, Codex or GitHub Copilot.
- Experience in a forward-deployed, solutions engineering, or internal tools role, or a background in operations, technical support, or site reliability engineering.
- Experience with Azure.
Familiarity with tools such as GitHub, Grafana, Jira, and Confluence.
*
About Chatham Financial:
Chatham Financial is the leading independent capital markets advisor, delivering an integrated blend of expert advice and powerful technology to help you reduce risk and seize opportunity. With decades of capital markets strategy, execution, monitoring, and performance expertise, we serve as an unwavering advocate for your best interests and your innovation partner. Our technology platform unifies data across assets, debt, and derivatives, giving you unmatched agility, transparency, and insight.
It’s clear ahead.
Our commitment is to carry that light forward in every partnership, every solution, and every market we serve.
We help guide the way—giving clients the insight and momentum to move forward with confidence, no matter what lies ahead.
Role Details
About This Role
AI Agent Developers build autonomous systems that can reason, plan, and take actions. They design multi-step workflows, tool-use frameworks, and orchestration layers that let LLMs interact with external systems. This is the frontier of applied AI engineering.
Agent development is where the most interesting (and hardest) problems in applied AI live right now. Making an LLM answer a question is straightforward. Making it reliably execute a 15-step workflow that involves calling APIs, reading databases, making decisions, and recovering from errors is an unsolved problem. You're building systems that have to work despite the fact that the underlying model is non-deterministic.
Across the 10,419 AI roles we're tracking, AI Agent Developer positions make up 1% of the market. At Chatham Financial, this role fits into their broader AI and engineering organization.
AI Agent Developer is one of the newest and fastest-growing AI role categories. The market is early but accelerating as companies move beyond simple chatbots toward AI systems that can take real actions. Compensation is high because the skill set is rare and the business impact is potentially enormous.
What the Work Looks Like
A typical week includes: designing the action space and tool definitions for a new agent use case, debugging why the agent chose the wrong action sequence on a specific input, building evaluation frameworks that test agent reliability across hundreds of scenarios, optimizing the prompt chain for cost and latency, and implementing safety guardrails to prevent the agent from taking destructive actions. The work is equal parts engineering and empirical science.
AI Agent Developer is one of the newest and fastest-growing AI role categories. The market is early but accelerating as companies move beyond simple chatbots toward AI systems that can take real actions. Compensation is high because the skill set is rare and the business impact is potentially enormous.
Skills Required
Deep experience with LLM APIs and agent frameworks (LangChain, CrewAI, AutoGen). Strong understanding of prompt engineering, function calling, and error handling for non-deterministic systems. Python is standard. Experience with orchestration patterns, state management, and workflow engines adds significant value.
The best agent developers think like systems engineers. They design for failure modes, build observability into every step, and understand that agent reliability is the product. Expertise in evaluation methodology for non-deterministic systems is the differentiator. Can you measure whether your agent works 'well enough'? Can you find the edge cases where it breaks?
Look for roles that describe specific agent use cases, mention evaluation methodology, and talk about production deployment. Early-stage companies exploring agents can be exciting, but be prepared for ambiguity. The most valuable roles are at companies that have already shipped a v1 and need to make it reliable.
Compensation Benchmarks
AI Agent Developer roles pay a median of $215,000 based on 252 positions with disclosed compensation. Mid-level AI roles across all categories have a median of $193,200.
Across all AI roles, the market median is $214,834. Top-quartile compensation starts at $266,000. The 90th percentile reaches $321,730. For comparison, the highest-paying categories include AI Safety ($274,500) and Research Engineer ($252,000). By seniority level: Entry: $114,400; Mid: $193,200; Senior: $230,000; Director: $272,100; VP: $235,000.
Chatham Financial AI Hiring
Chatham Financial has 1 open AI role right now. They're hiring across AI Agent Developer. Based in Kennett Square, PA, US.
Location Context
Across all AI roles, 6% (580 positions) offer remote work, while 9,776 require on-site attendance. Top AI hiring metros: New York (3,785 roles, $222,480 median); San Francisco (3,262 roles, $261,300 median); Los Angeles (1,896 roles, $210,000 median).
Career Path
Common paths into AI Agent Developer roles include Software Engineer, LLM Engineer, Prompt Engineer.
From here, career progression typically leads toward AI Architect, Principal Engineer, Head of AI Engineering.
Build agents. That's the portfolio. Take an open-source agent framework, build something that completes a non-trivial multi-step task, evaluate it rigorously, and document what you learned about reliability, cost, and failure modes. The field is new enough that practical experience counts for more than credentials.
What to Expect in Interviews
Interviews focus on systems thinking and reliability engineering. Expect questions about agent architecture: how you'd design a multi-step workflow with error recovery, how you'd evaluate agent performance, and how you'd prevent agents from taking destructive actions. Coding exercises often involve building a simple agent with tool use and evaluating its behavior across different scenarios. Discussion of safety and guardrails is increasingly common.
When evaluating opportunities: Look for roles that describe specific agent use cases, mention evaluation methodology, and talk about production deployment. Early-stage companies exploring agents can be exciting, but be prepared for ambiguity. The most valuable roles are at companies that have already shipped a v1 and need to make it reliable.
AI Hiring Overview
The AI job market has 10,419 open positions tracked in our dataset. By seniority: 117 entry-level, 4,975 mid-level, 3,840 senior, and 1,487 leadership roles (Director, VP, C-Level). Remote roles make up 6% of the market (580 positions). The remaining 9,776 roles require on-site or hybrid attendance.
The market median for AI roles is $214,834. Top-quartile compensation starts at $266,000. The 90th percentile reaches $321,730. Highest-paying categories: AI Safety ($274,500 median, 90 roles); Research Engineer ($252,000 median, 580 roles); AI Engineering Manager ($250,811 median, 72 roles).
AI Agent Developer is one of the newest and fastest-growing AI role categories. The market is early but accelerating as companies move beyond simple chatbots toward AI systems that can take real actions. Compensation is high because the skill set is rare and the business impact is potentially enormous.
The AI Job Market Today
The AI job market spans 10,419 open positions across 16 role categories. The largest categories by volume: AI/ML Engineer (7,849), AI Product Manager (516), AI Software Engineer (513). These three account for the majority of open positions, though smaller categories often have higher per-role compensation because of specialized skill requirements.
The seniority mix tells a story about where AI teams are in their maturity. Entry-level roles (117) are outnumbered by mid-level (4,975) and senior (3,840) positions, reflecting that most companies are past the 'build a team from scratch' phase and need experienced engineers who can ship production systems. Leadership roles (Director, VP, C-Level) total 1,487 positions, representing the bottleneck between technical execution and organizational strategy.
Remote work availability sits at 6% of all AI roles (580 positions), with 9,776 requiring on-site or hybrid attendance. The remote share has stabilized after the post-pandemic correction. Senior and specialized roles (Research Scientist, ML Architect) are more likely to be remote-eligible than entry-level positions, partly because experienced hires have more negotiating power and partly because these roles require less hands-on mentorship.
AI compensation is structured in clear tiers. The market median sits at $214,834. Top-quartile roles start at $266,000, and the 90th percentile reaches $321,730. These figures include base salary with disclosed compensation. Total compensation (including equity, bonuses, and sign-on) runs 20-40% higher at companies that offer those components.
Category matters for compensation. AI Safety roles lead at $274,500 median, while Prompt Engineer roles sit at $140,000. The spread between highest and lowest-paying categories reflects the premium on specialized technical skills versus broader analytical roles.
The most in-demand skills across all AI postings: Python (2,350 postings), Aws (1,173 postings), Azure (1,030 postings), Rag (986 postings), Gcp (714 postings), Pytorch (649 postings), Kubernetes (583 postings), Claude (581 postings). Python dominates, appearing in the vast majority of role descriptions regardless of category. Cloud platform experience (AWS, GCP, Azure) is the second most common requirement. The newer entrants to the top skills list (RAG, vector databases, LLM APIs) reflect the shift from traditional ML toward generative AI applications.
Frequently Asked Questions
Get Weekly AI Career Intelligence
Salary data, skills demand, and market signals from 16,000+ AI job postings. Every Monday.