As a Senior Data Engineer, CASM Platform, you will:
Data Integration, API Development:  Integrate diverse cybersecurity data sources using a variety of API mechanisms and standardize and streamline data across the data and user planes.
Build and maintain data APIs:  Enable seamless access to data pipelines, allowing real-time insights for applications, machine learning models, and analytical layers.
Data Pipeline Engineering Optimization:  Design, develop, and optimize large-scale ETL/ELT pipelines on Databricks to efficiently process and transform cybersecurity data. Utilize Python, PySpark, and Databricks to automate and standardize data workflows across stages (raw, cleaned, curated), ensuring scalability and high performance.
Data Quality & Governance:  Implement automated data quality checks, leveraging Databricks DQM tools and CI/CD pipelines to uphold data integrity and governance standards. Ensure data lineage, metadata management, and compliance with cybersecurity and privacy regulations, applying rigorous quality standards across data ingestion and processing workflows.
Data Analytics & Visualization:  Design centralized data models and perform in-depth data analysis to support cybersecurity and risk management objectives. Develop visualizations and dashboards using tools like Databricks, encapsulating data to spin up to the React.js application layer to provide stakeholders with actionable insights into threat landscapes, vulnerability trends, and performance metrics across the platform.
Scalable & Secure Data Architecture:  Architect and manage secure, high-performance data environments on Databricks, utilizing AWS services such as S3, ELB, and Lambda. Ensure data availability, consistency, and security, aligning with AWS best practices and data encryption standards to safeguard sensitive cybersecurity data.
Agile Product & Engineering Continuous Delivery:  Collaborate with advanced Agile Product and Engineering cross-functional teams to deliver data-driven insights through analytics tools and custom visualizations that inform strategy and decision-making. Empower stakeholders with timely, actionable intelligence from complex data analyses, enhancing their ability to respond to evolving cybersecurity risks.
Data Science & ML Integration:  Deploy machine learning models, including predictive analytics, anomaly detection, and risk scoring algorithms, into the CASM platform. Leverage Python and PySpark to enable real-time and batch processing of model outputs, enhancing CASM Platform’s proactive threat detection and response capabilities.
Mentorship & Best Practices Promotion:  Mentor junior engineers, establishing best practices in data engineering, DevOps, data science, and analytics. Encourage high standards in model deployment, data security, performance optimization, and visualization practices, fostering a culture of innovation and excellence.
Education Qualifications:
Minimum Qualifications:
Education: B.S., M.S., or Ph.D. in Computer Science, Data Science, Information Systems, or a related field, or equivalent professional experience.
Technical Expertise: 8+ years in data engineering with strong skills in Python, PySpark, SQL, and extensive, hands-on experience with Databricks and big data frameworks. Expertise in integrating data science workflows and deploying ML models for real-time and batch processing within a cybersecurity context.
Cloud Proficiency: Advanced proficiency in AWS, including EC2, S3, Lambda, ELB, and container orchestration (Docker, Kubernetes). Experience in managing large-scale data environments on AWS, optimizing for performance, security, and compliance.
Security Integration: Proven experience implementing SCAS, SAST, DAST/WAS, and secure DevOps practices within an SDLC framework to ensure data security and compliance in a high-stakes cybersecurity environment.
Data Architecture: Demonstrated ability to design and implement complex data architectures, including data lakes, data warehouses, and lake house solutions. Emphasis on secure, scalable, and highly available data structures that support ML-driven insights and real-time analytics.
Data Quality & Governance: Hands-on experience with automated data quality checks, data lineage, and governance standards. Proficiency in Databricks DQM or similar tools to enforce data integrity and compliance across pipelines.
Data Analytics & Visualization: Proficiency with analytics and visualization tools such as Databricks, Power BI, and Tableau to generate actionable insights for cybersecurity risks, threat patterns, and vulnerability trends. Skilled in translating complex data into accessible visuals and reports for cross-functional teams.
CI/CD and Automation: Experience building CI/CD pipelines that automate testing, security scans, and deployment processes. Proficiency in deploying ML models and data processing workflows using CI/CD, ensuring consistent quality and streamlined delivery.
Agile Experience: Deep experience in Agile/Scrum environments, with a thorough understanding of Agile core values and principles, effectively delivering complex projects with agility and cross-functional collaboration.
Preferred Experience:
Advanced Data Modeling & Governance: Expertise in designing data models for cybersecurity data analytics, emphasizing data lineage, federation, governance, and compliance. Experience ensuring security and privacy within data architectures.
Machine Learning & Predictive Analytics: Experience deploying ML algorithms, predictive models, and anomaly detection frameworks to bolster CASM platform’s cybersecurity capabilities.
High-Performance Engineering Culture: Background in mentoring engineers in data engineering best practices, promoting data science, ML, and analytics integration, and fostering a culture of collaboration and continuous improvement.
Department: Preferred Vendors
This is a contract position.

#J-18808-Ljbffr

Job Title:  Azure Data Engineer
Location:  Spring Texas (On-Site)
Need Only Local to Houston, TX
Experience:  10+ Years
Responsibilities     Perform ETL, ELT operations and administration using modern tools, programming languages and systems securely and in accordance with enterprise data standards.
Assemble, model, and transform large complex sets of data that meet non-functional and functional business requirements into a format that can be analyzed.
Automate data processing of data from multiple data sources.
Develop, deploy and version control code for data consumption, reuse for APIs.
Employ machine learning techniques to create and sustain data structures.
Perform root cause analysis on external and internal processes and data to identify opportunity for improvement, resolve data quality issues.
Lead data-related workshops with stakeholders to capture data requirements and acceptance criteria.

Skills and Qualifications   We are looking for someone who has technical depth and broad tool experience related to data engineering and has the following required qualifications:
Minimum bachelor’s degree in: Data Science, Business Intelligence, Statistics, Computer Engineering or related field, or the equivalent combination of education, professional training, and work experience.
5 or more years performing duties related to data engineering.
Advanced English level.
Expert proficiency in at least one of these programming languages: Python, NoSQL, SQL, R, and competent in source code management.
Build processes supporting data transformation, data structures, metadata, dependency, and workload management.
Create data validation methods and data analysis tools.

Preferred Qualifications / Experience     Excellent problem-solving skills and ability to learn through scattered resources.
Automate routine tasks via scripts, code.
Capacity to successfully manage a pipeline of duties with minimal supervision.
Experience supporting and working with cross-functional teams in a dynamic environment.
Modify existing reports, extracts, dashboards, and cubes as necessary.
Commitment to operations integrity and ability to hold self and others accountable for results.

Required Proficiency:     Advanced: SQL, Snowflake, Tamr, Python, GitHub
Intermediate: Databricks, Airflow

#J-18808-Ljbffr

As a Senior Data Engineer, CASM Platform, you will:
Data Integration, API Development: Integrate diverse cybersecurity data sources using a variety of API mechanisms and standardize and streamline data across the data and user planes.
Build and maintain data APIs for seamless access to data pipelines, enabling real-time insights for applications, machine learning models, and analytical layers.
Data Pipeline Engineering Optimization: Design, develop, and optimize large-scale ETL/ELT pipelines on Databricks to efficiently process and transform cybersecurity data. Utilize Python, PySpark, and Databricks to automate and standardize data workflows across stages (raw, cleaned, curated), ensuring scalability and high performance.
Data Quality & Governance: Implement automated data quality checks, leveraging Databricks DQM tools and CI/CD pipelines to uphold data integrity and governance standards. Ensure data lineage, metadata management, and compliance with cybersecurity and privacy regulations, applying rigorous quality standards across data ingestion and processing workflows.
Data Analytics & Visualization: Design centralized data models and perform in-depth data analysis to support cybersecurity and risk management objectives. Develop visualizations and dashboards using tools like Databricks, encapsulating data to spin up to React.js application layer to provide stakeholders with actionable insights into threat landscapes, vulnerability trends, and performance metrics across the platform.
Scalable & Secure Data Architecture: Architect and manage secure, high-performance data environments on Databricks, utilizing AWS services such as S3, ELB, and Lambda. Ensure data availability, consistency, and security, aligning with AWS best practices and data encryption standards to safeguard sensitive cybersecurity data.
Agile Product; Engineering Continuous Delivery: Collaborate with advanced Agile Product; Engineering cross-functional teams to deliver data-driven insights through analytics tools and custom visualizations that inform strategy and decision-making. Empower stakeholders with timely, actionable intelligence from complex data analyses, enhancing their ability to respond to evolving cybersecurity risks.
Data Science & ML Integration: Deploy machine learning models, including predictive analytics, anomaly detection, and risk scoring algorithms, into the CASM platform. Leverage Python and PySpark to enable real-time and batch processing of model outputs, enhancing CASM Platform’s proactive threat detection and response capabilities.
Mentorship; Best Practices Promotion: Mentor junior engineers, establishing best practices in data engineering, DevOps, data science, and analytics. Encourage high standards in model deployment, data security, performance optimization, and visualization practices, fostering a culture of innovation and excellence.
Education Qualifications:
Minimum Qualifications
• Education: B.S., M.S., or Ph.D. in Computer Science, Data Science, Information Systems, or a related field, or equivalent professional experience.
• Technical Expertise: 8+ years in data engineering with strong skills in Python, PySpark, SQL, and extensive, hands-on experience with Databricks and big data frameworks. Expertise in integrating data science workflows and deploying ML models for real-time and batch processing within a cybersecurity context.
• Cloud Proficiency: Advanced proficiency in AWS, including EC2, S3, Lambda, ELB, and container orchestration (Docker, Kubernetes). Experience in managing large-scale data environments on AWS, optimizing for performance, security, and compliance.
• Security Integration: Proven experience implementing SCAS, SAST, DAST/WAS, and secure DevOps practices within an SDLC framework to ensure data security and compliance in a high-stakes cybersecurity environment.
• Data Architecture: Demonstrated ability to design and implement complex data architectures, including data lakes, data warehouses, and lake house solutions. Emphasis on secure, scalable, and highly available data structures that support ML-driven insights and real-time analytics.
• Data Quality & Governance: Hands-on experience with automated data quality checks, data lineage, and governance standards. Proficiency in Databricks DQM or similar tools to enforce data integrity and compliance across pipelines.
• Data Analytics & Visualization: Proficiency with analytics and visualization tools such as Databricks, Power BI, and Tableau to generate actionable insights for cybersecurity risks, threat patterns, and vulnerability trends. Skilled in translating complex data into accessible visuals and reports for cross-functional teams.
• CI/CD and Automation: Experience building CI/CD pipelines that automate testing, security scans, and deployment processes. Proficiency in deploying ML models and data processing workflows using CI/CD, ensuring consistent quality and streamlined delivery.
• Agile Experience: Deep experience in Agile/Scrum environments, with a thorough understanding of Agile core values and principles, effectively delivering complex projects with agility and cross-functional collaboration.
Preferred Experience:
• Advanced Data Modeling & Governance: Expertise in designing data models for cybersecurity data analytics, emphasizing data lineage, federation, governance, and compliance. Experience ensuring security and privacy within data architectures.
• Machine Learning & Predictive Analytics: Experience deploying ML algorithms, predictive models, and anomaly detection frameworks to bolster CASM platform’s cybersecurity capabilities.
• High-Performance Engineering Culture: Background in mentoring engineers in data engineering best practices, promoting data science, ML, and analytics integration, and fostering a culture of collaboration and continuous improvement.
#J-18808-Ljbffr

Data Engineer Cedent Jobs

Search for Job title, position, company and/or location…

Senior Data Engineer
April 5, 2025

Azure Data Engineer Houston Tx
March 23, 2025

Senior Data Engineer Boston Ma
March 22, 2025

Quick filters

Top companies