Staff Agentic Data Engineer – Data Platform

Remote Senior Data Engineer

Interested in this Data Engineer role at Nucleus Security?

Apply Now →

Skills & Technologies

Python

About This Role

AI job market dashboard showing open roles by category

Are you looking for more in life than just building another data pipeline? Does upending cybersecurity resonate with you? We're a rapidly expanding cybersecurity company revolutionizing vulnerability management for organizations of all sizes. For our customers, vulnerability management has historically been a game of catch-up, with fragmented data, limited asset coverage, and manual processes.

Nucleus' primary mission is to create a fast, scalable platform that brings together large volumes of security data and transforms it into actionable insights, making vulnerability management simple, efficient, and effective.

We're looking for a Staff / Principal Data Engineer to help define and evolve the architecture behind our data platform. This is a senior technical leadership role for an engineer who thrives on solving complex data problems at scale, designing systems that process large and growing datasets, and establishing patterns that other engineers can build upon.

What You Will Do

  • Own the architecture and evolution of large-scale data processing and analytics systems, from ingestion and normalization through transformation, aggregation, and customer-facing analytics.
  • Design and evolve data models that support high-volume analytical workloads while balancing query performance, storage efficiency, maintainability, and scalability.
  • Design and optimize complex analytical SQL queries operating across large datasets, identifying performance bottlenecks and developing scalable solutions.
  • Architect distributed and asynchronous job processing systems capable of reliably processing large volumes of data across concurrent workloads.
  • Design and evolve Spark, Flink or similar technologies based data processing workflows for large-scale transformation, enrichment, aggregation, and computation.
  • Drive the effective use of SingleStore/MemSQL, including schema design, indexing and partitioning strategies, query optimization, and workload design.
  • Establish architectural patterns for batch processing, data transformation, job orchestration, failure recovery, idempotency, and reprocessing.
  • Diagnose complex performance and scalability issues across databases, distributed processing systems, application services, and infrastructure.
  • Partner with engineering and product teams to translate ambiguous analytical and data-processing requirements into scalable technical architectures.
  • Define technical standards and reusable patterns for data modeling, data processing, and analytical query development.
  • Provide technical leadership across engineering initiatives, influencing architecture and mentoring engineers while remaining deeply hands-on with implementation.
  • Identify architectural limitations before they become operational problems and lead improvements that increase the scalability, reliability, and performance of the platform.

The Following Skills and Experience Are Key to Succeeding in This Role

  • 10+ years of professional software engineering, data engineering, or data platform experience, with significant experience designing and operating production data systems.
  • Deep expertise in relational and analytical data modeling, including designing schemas for large-scale analytical and reporting workloads.
  • Expert-level SQL, with demonstrated experience designing, analyzing, and optimizing complex analytical queries over large datasets.
  • Strong experience with SingleStore/MemSQL or comparable distributed SQL/analytical database technologies.
  • Significant hands-on experience with Spark, Flink or similar technologies, including designing and optimizing distributed data-processing workloads.
  • Strong understanding of distributed data processing, including partitioning, parallelism, data skew, resource utilization, and performance optimization.
  • Experience designing high-throughput job processing systems, including scheduling, concurrency management, retries, failure recovery, idempotency, and observability.
  • Experience designing data pipelines that support batch ingestion, transformation, enrichment, aggregation, and downstream analytics.
  • Strong understanding of database performance characteristics, including indexing, partitioning, query execution plans, data distribution, and memory/compute trade-offs.
  • Experience identifying and resolving performance bottlenecks across application, database, and data-processing layers.
  • Proficiency in Python, Java or another production programming language used to build data-processing and distributed systems.
  • Strong understanding of testing strategies for data-intensive systems, including unit, integration, data-quality, and end-to-end testing.
  • Proven ability to independently own technically ambiguous problems and drive them from investigation through production implementation.
  • Demonstrated ability to influence architecture and technical direction across multiple teams without relying solely on organizational authority.
  • Ability to communicate complex data architecture and performance trade-offs clearly to engineers, product stakeholders, and technical leadership.
  • Ability to collaborate effectively with distributed teams across multiple time zones.

Preferred Qualifications

  • Deep production experience with SingleStore/MemSQL, particularly with large-scale analytical workloads.
  • Advanced experience with pipeline performance tuning, partitioning strategies, memory management, and optimization of large distributed workloads.
  • Experience designing platforms that support a combination of high-volume ingestion and low-latency analytical queries.
  • Experience building or operating distributed job execution and orchestration systems.
  • Experience with large-scale cybersecurity, vulnerability, asset, telemetry, or event datasets.
  • Experience working with vulnerability scanning technologies across the security stack, including SCA, SAST, DAST, IAST, vulnerability scanning, and container security.
  • Experience designing systems for incremental processing, backfills, replay/reprocessing, deduplication, and late-arriving data.
  • Experience with data observability and operational monitoring for large-scale processing workloads.
  • Experience integrating AI/LLM-driven capabilities with data platforms, including preparing and processing large datasets for AI-assisted analytical workflows.
  • Experience operating data-intensive applications on Linux and cloud infrastructure.
  • Familiarity with Agile/Scrum methodologies in a professional engineering environment.

Why You Should Be Excited

  • Nucleus is a truly unique solution that’s defining a market AND making an actual impact.
  • We’re biased, but you will get to work with one of the best teams in security. We have a lot to get done and we work extremely hard, but we have fun in the process.
  • This role has the opportunity to directly influence how Nucleus approaches data architecture, analytical processing, and scalability as the platform and the datasets it processes continue to grow.
  • Outstanding benefits include flexible PTO, 401k company contribution (not a match, a contribution), generous education and training budget, 100% company-paid healthcare, fully remote work, and more.

Additional Information

At Nucleus we are committed to achieving excellence in our field by combining diversity, collaboration, teamwork, and pride in our work. All qualified applicants will receive consideration for employment without regard to race, sex, color, religion, sexual orientation, gender identity, national origin, protected veteran status, or disability.

0Kz8RFQBnU

Role Details

Title Staff Agentic Data Engineer – Data Platform
Location Remote, US
Category Data Engineer
Experience Senior
Salary Not disclosed
Remote Yes

About This Role

Data Engineers build the pipelines that feed AI models. They design ETL workflows, manage data lakes, and ensure training and inference data is clean, timely, and accessible. Without good data engineering, AI projects fail. It's that simple.

The AI era has expanded the data engineer's scope far beyond batch ETL jobs. You're building real-time embedding pipelines for RAG systems, managing vector databases, ensuring training data quality at scale, and building the infrastructure that lets ML teams iterate on data as fast as they iterate on models. Data quality is the biggest predictor of model quality, and you're the person responsible for it.

Across the 10,419 AI roles we're tracking, Data Engineer positions make up 1% of the market. At Nucleus Security, this role fits into their broader AI and engineering organization.

Data Engineer demand in AI contexts is strong and growing. Every company building AI needs clean, reliable data pipelines. The shift toward real-time AI applications (chatbots, recommendation engines, agent systems) means data engineering is more critical than ever. Companies are willing to pay premium salaries for data engineers with AI/ML pipeline experience.

What the Work Looks Like

A typical week includes: debugging a data pipeline that's producing stale embeddings for the RAG system, optimizing a Spark job that processes training data, building a data quality monitoring dashboard, meeting with the ML team to understand their next data requirements, and writing dbt models that transform raw event data into ML-ready features. The work is deeply technical and high-impact.

Data Engineer demand in AI contexts is strong and growing. Every company building AI needs clean, reliable data pipelines. The shift toward real-time AI applications (chatbots, recommendation engines, agent systems) means data engineering is more critical than ever. Companies are willing to pay premium salaries for data engineers with AI/ML pipeline experience.

Skills Required

Python (23% of roles)

SQL, Python, and distributed systems (Spark, Airflow, dbt) are core. Cloud data platforms (Snowflake, BigQuery, Redshift) are increasingly standard. Many AI-focused roles also want familiarity with vector databases and embedding pipelines. Understanding data modeling, pipeline orchestration, and data quality frameworks covers the essentials.

AI-specific data engineering skills include: building feature stores, managing training data versioning, implementing data lineage tracking, and building real-time embedding pipelines. Experience with streaming systems (Kafka, Flink) is valuable for real-time AI applications. Understanding ML data requirements (balanced datasets, data augmentation, evaluation set construction) makes you much more effective working with ML teams.

Strong postings specify the data stack, mention ML pipeline work, and describe the scale of data you'll be working with. Look for companies that understand the connection between data quality and model quality. Avoid roles that conflate data engineering with data analysis.

Compensation Benchmarks

Data Engineer roles pay a median of $178,650 based on 241 positions with disclosed compensation. Senior-level AI roles across all categories have a median of $230,000.

Across all AI roles, the market median is $214,834. Top-quartile compensation starts at $266,000. The 90th percentile reaches $321,730. For comparison, the highest-paying categories include AI Safety ($274,500) and Research Engineer ($252,000). By seniority level: Entry: $114,400; Mid: $193,200; Senior: $230,000; Director: $272,100; VP: $235,000.

Nucleus Security AI Hiring

Nucleus Security has 2 open AI roles right now. They're hiring across AI/ML Engineer, Data Engineer. Based in Remote, US.

Remote Work Context

Remote AI roles pay a median of $188,000 across 2,811 positions. About 6% of all AI roles offer remote work.

Career Path

Common paths into Data Engineer roles include Backend Engineer, Database Administrator, Analytics Engineer.

From here, career progression typically leads toward Senior Data Engineer, ML Engineer, Data Platform Lead.

Master SQL and Python first. Then learn a distributed processing framework (Spark or its modern alternatives) and a pipeline orchestrator (Airflow, Dagster, Prefect). Build a portfolio project that demonstrates end-to-end pipeline construction: ingest, transform, validate, serve. If you want to specialize in AI data engineering, add vector databases and embedding pipelines to your skill set.

What to Expect in Interviews

Expect SQL deep-dives (query optimization, partitioning strategies, data modeling), Python coding focused on data pipeline patterns, and system design questions about building scalable ETL workflows. Companies with ML teams will ask about feature stores, embedding pipelines, and training data management. Be ready to discuss data quality monitoring, pipeline orchestration, and how you'd handle schema evolution in a production data lake.

When evaluating opportunities: Strong postings specify the data stack, mention ML pipeline work, and describe the scale of data you'll be working with. Look for companies that understand the connection between data quality and model quality. Avoid roles that conflate data engineering with data analysis.

AI Hiring Overview

The AI job market has 10,419 open positions tracked in our dataset. By seniority: 117 entry-level, 4,975 mid-level, 3,840 senior, and 1,487 leadership roles (Director, VP, C-Level). Remote roles make up 6% of the market (580 positions). The remaining 9,776 roles require on-site or hybrid attendance.

The market median for AI roles is $214,834. Top-quartile compensation starts at $266,000. The 90th percentile reaches $321,730. Highest-paying categories: AI Safety ($274,500 median, 90 roles); Research Engineer ($252,000 median, 580 roles); AI Engineering Manager ($250,811 median, 72 roles).

Data Engineer demand in AI contexts is strong and growing. Every company building AI needs clean, reliable data pipelines. The shift toward real-time AI applications (chatbots, recommendation engines, agent systems) means data engineering is more critical than ever. Companies are willing to pay premium salaries for data engineers with AI/ML pipeline experience.

The AI Job Market Today

The AI job market spans 10,419 open positions across 16 role categories. The largest categories by volume: AI/ML Engineer (7,849), AI Product Manager (516), AI Software Engineer (513). These three account for the majority of open positions, though smaller categories often have higher per-role compensation because of specialized skill requirements.

The seniority mix tells a story about where AI teams are in their maturity. Entry-level roles (117) are outnumbered by mid-level (4,975) and senior (3,840) positions, reflecting that most companies are past the 'build a team from scratch' phase and need experienced engineers who can ship production systems. Leadership roles (Director, VP, C-Level) total 1,487 positions, representing the bottleneck between technical execution and organizational strategy.

Remote work availability sits at 6% of all AI roles (580 positions), with 9,776 requiring on-site or hybrid attendance. The remote share has stabilized after the post-pandemic correction. Senior and specialized roles (Research Scientist, ML Architect) are more likely to be remote-eligible than entry-level positions, partly because experienced hires have more negotiating power and partly because these roles require less hands-on mentorship.

AI compensation is structured in clear tiers. The market median sits at $214,834. Top-quartile roles start at $266,000, and the 90th percentile reaches $321,730. These figures include base salary with disclosed compensation. Total compensation (including equity, bonuses, and sign-on) runs 20-40% higher at companies that offer those components.

Category matters for compensation. AI Safety roles lead at $274,500 median, while Prompt Engineer roles sit at $140,000. The spread between highest and lowest-paying categories reflects the premium on specialized technical skills versus broader analytical roles.

The most in-demand skills across all AI postings: Python (2,350 postings), Aws (1,173 postings), Azure (1,030 postings), Rag (986 postings), Gcp (714 postings), Pytorch (649 postings), Kubernetes (583 postings), Claude (581 postings). Python dominates, appearing in the vast majority of role descriptions regardless of category. Cloud platform experience (AWS, GCP, Azure) is the second most common requirement. The newer entrants to the top skills list (RAG, vector databases, LLM APIs) reflect the shift from traditional ML toward generative AI applications.

Frequently Asked Questions

Based on 241 roles with disclosed compensation, the median salary for Data Engineer positions is $178,650. Actual compensation varies by seniority, location, and company stage.
SQL, Python, and distributed systems (Spark, Airflow, dbt) are core. Cloud data platforms (Snowflake, BigQuery, Redshift) are increasingly standard. Many AI-focused roles also want familiarity with vector databases and embedding pipelines. Understanding data modeling, pipeline orchestration, and data quality frameworks covers the essentials.
About 6% of the 10,419 AI roles we track offer remote work. Remote availability varies by company and seniority level, with senior and leadership roles more likely to offer location flexibility.
Nucleus Security is among the companies actively hiring for AI and ML talent. Check our company profiles for detailed breakdowns of open roles, salary ranges, and hiring trends.
Common next steps from Data Engineer positions include Senior Data Engineer, ML Engineer, Data Platform Lead. Progression depends on whether you lean toward technical depth, people management, or product strategy.

Get Weekly AI Career Intelligence

Salary data, skills demand, and market signals from 16,000+ AI job postings. Every Monday.