Principal Data Engineer/ Architect Temp to perm Full time - Pay $70 ot $100/hr based on experience Remote As the Senior Software Engineer, you will lead a team of data engineers in designing, building, and maintaining high-performance software system to manage analytical data pipelines that fuel the organization's data strategy using software engineering best practices. Beyond technical expertise, you will also serve as a change leader, guiding teams through adopting new tools, technologies, and workflows to improve data management and processing. This position requires extensive hands-on data system design and coding experience, as well as the development of modern data pipelines (AWS Step functions, Prefect, Airflow, Luigi, Python, Spark, SQL) and associated code in AWS. You will work closely with stakeholders across the business to understand their data needs, ensure scalability, and foster a culture of innovation and learning within the data engineering team and beyond. Key Responsibilities: •Be responsible for the overall architecture of a specific module within a product (e.g., Data-ingestion, near-real-time-data-processor, etc.), perform design and assist implementation considering system characteristics to produce optimal performance, reliability and maintainability. •Provide technical guidance to team members, ensuring they are working towards the product's architectural goals. •Create and manage RFCs (Request for Comments) and ADRs (Architecture Decision Records), Design notes and technical documentation for your module, following the architecture governance processes. •Lead a team of data engineers, providing mentorship, setting priorities, and ensuring alignment with business goals. •Architect, design, and build scalable data pipelines for processing large volumes of structured and unstructured data from various sources. •Collaborate with software engineers, architects, and product teams to design and implement systems that enable real-time and batch data processing at scale. •Be the go-to person for PySpark-based solutions, ensuring optimal performance and reliability for distributed data processing. •Ensure that data engineering systems adhere to the best data security, privacy, and governance practices in line with industry standards. •Perform code reviews for the product, ensuring adherence to company coding standards and best practices. •Develop and implement monitoring and alerting systems to ensure timely detection and resolution of data pipeline failures and performance bottlenecks. •Act as a champion for new technologies, helping ease transitions and addressing concerns or resistance from team members. Ideal Candidate: •Experience leading a data engineering team with a strong focus on software engineering principles such as KISS, DRY, YAGNI etc. •Candidate MUST have experience in owning large, complex system architecture and hands-on experience designing and implementing data pipelines across large-scale systems. •Experience implementing and optimizing data pipelines with AWS is a must. •Production delivery experience in Cloud-based PaaS Big Data related technologies (EMR, Snowflake, Data bricks etc.) •Experienced in multiple Cloud PaaS persistence technologies, and in-depth knowledge of cloud- based ETL offerings and orchestration technologies (AWS Step Function, Airflow etc.) •Experienced in stream-based and batch processing, applying modern technologies •Working experience with distributed file systems (S3, HDFC, ADLS), table formats (HUDI, Iceberg), and various open file formats (JSON, Parquet, Csv, etc.) •Strong programming experience in PySpark, SQL, Python, etc. •Database design skills including normalization/de-normalization and data warehouse design •Knowledge and understanding of relevant legal and regulatory requirements, such as SOX, PCI, HIPAA, Data Protection •Experience in the healthcare industry, a plus •A collaborative and informative mentality is a must! Toolset: •AWS, preferably AWS certified Data Engineer and AWS certified Solutions Architect. •Proficiency in at least one programming language C#, GoLang, JavaScript or ReactJs •Spark / Python / SQL •Snowflake/ Databricks / Synapse / MS SQL Server •ETL / Orchestration Tools (Step Function, DBT etc.) •ML / Notebooks Education and experience required •Bachelors or Master's in Computer Science, Information Systems, or an engineering field or relevant experience. •10+ years of related experience in developing data solutions and data movement. This role can be REMOTE

<h3>Company Description</h3>In business since 2000, LanceSoft is a nationally certified Minority Business Enterprise (MBE) that has established itself as a pioneer in providing highly scalable, cost-effective workforce solutions to a diverse set of customers across various industries in the United States. Headquartered in Virginia, LanceSoft currently operates out of several locations in the US and Canada.<h3>Job Description</h3>The Sr. Data Engineer will be responsible for assisting business teams with data analytics needs, working with Global Services Data Architect to make sense out of the vast amount of machine data collected over time.Duties include (but are not limited to):<ol><li>Application development - ensure translation of requirements into a logical and physical database architecture/design needs</li><li>Application development – participate in cross-functional and global application teams to design and develop data layers and data stores (data services in Java, stored procedures in Oracle)</li><li>Application development – educate and mentor peers and application teams on best practices on data management</li><li>Production support – database performance monitoring, troubleshooting, and tuning</li><li>Data analytics – explore data sets and identify data transformation and data quality needs for targeted applications</li><li>Data analytics – implement known data engineering tasks including transformation, quality remediation and report, audit, optimization for storage and retrieval</li><li>Data analytics – develop analytics / algorithms / rules to derive insight and business value</li></ol>Quality Specific Goals:<ol><li>Aware of and comply with the GEHC Quality Manual, Quality Management System, Quality Management Policy, Quality Goals, and applicable laws and regulations as they apply to this job type/position</li><li>Complete all planned Quality & Compliance training within the defined deadlines</li><li>Identify and report any quality or compliance concerns and take immediate corrective action as required</li><li>Ensure compliance/closure of Regulatory and Quality requirements before approving Design Outputs/Program Deliverables</li><li>Lead continuous improvement activities by driving the implementation of process and product quality improvement initiatives</li></ol><h3>Qualifications</h3>• Bachelor degree in Computer Science or Engineering, or closely related fields• 4+ years of experience designing and implementing SQL queries, stored procedures, and optimizing their performance• 2+ years of experience working on Oracle data administration• Experience in software development and the software development lifecycle• Experience with batch processing and large databases• Expertise in one or more analytics software tools and packages• Familiar with data modeling and physical database design• Familiar with Unix• Strong communication (written and verbal) and problem-solving skills• Ability to work independently, with limited informationPreferred Qualifications:• Familiar with Agile software development methodology• Experienced with developing with Java• Healthcare Services domain knowledge• Experience/familiarity with Remote Monitoring & Diagnostics• Basic knowledge of project planning and execution• Demonstrated experience with regulatory compliance process, and quality management process in the context of software development• Experience in working with global teams and users across the world
#J-18808-Ljbffr

Education & Qualifications
Minimum Qualifications
• Education: B.S., M.S., or Ph.D. in Computer Science, Data Science, Information Systems, or a related field, or equivalent professional experience.
• Technical Expertise: 8+ years in data engineering with strong skills in Python, PySpark, SQL, and extensive, hands-on experience with Databricks and big data frameworks. Expertise in integrating data science workflows and deploying ML models for real-time and batch processing within a cybersecurity context.
• Cloud Proficiency: Advanced proficiency in AWS, including EC2, S3, Lambda, ELB, and container orchestration (Docker, Kubernetes). Experience in managing large-scale data environments on AWS, optimizing for performance, security, and compliance.
• Security Integration: Proven experience implementing SCAS, SAST, DAST/WAS, and secure DevOps practices within an SDLC framework to ensure data security and compliance in a high-stakes cybersecurity environment.
• Data Architecture: Demonstrated ability to design and implement complex data architectures, including data lakes, data warehouses, and lake house solutions. Emphasis on secure, scalable, and highly available data structures that support ML-driven insights and real-time analytics.
• Data Quality & Governance: Hands-on experience with automated data quality checks, data lineage, and governance standards. Proficiency in Databricks DQM or similar tools to enforce data integrity and compliance across pipelines.
• Data Analytics & Visualization: Proficiency with analytics and visualization tools such as Databricks, Power BI, and Tableau to generate actionable insights for cybersecurity risks, threat patterns, and vulnerability trends. Skilled in translating complex data into accessible visuals and reports for cross-functional teams.
• CI/CD and Automation: Experience building CI/CD pipelines that automate testing, security scans, and deployment processes. Proficiency in deploying ML models and data processing workflows using CI/CD, ensuring consistent quality and streamlined delivery.
• Agile Experience: Deep experience in Agile/Scrum environments, with a thorough understanding of Agile core values and principles, effectively delivering complex projects with agility and cross-functional collaboration. Preferred Experience
• Advanced Data Modeling & Governance: Expertise in designing data models for cybersecurity data analytics, emphasizing data lineage, federation, governance, and compliance. Experience ensuring security and privacy within data architectures.
• Machine Learning & Predictive Analytics: Experience deploying ML algorithms, predictive models, and anomaly detection frameworks to bolster CASM platform's cybersecurity capabilities.
• High-Performance Engineering Culture: Background in mentoring engineers in data engineering best practices, promoting data science, ML, and analytics integration, and fostering a culture of collaboration and continuous improvement.

Pay: $80-90/hr
As a  Senior Data Engineer,
Education & Qualifications
Education:  B.S., M.S., or Ph.D. in Computer Science, Data Science, Information Systems, or a related field, or equivalent professional experience.
Technical Expertise:  8+ years in data engineering with strong skills in Python, PySpark, SQL, and extensive, hands-on experience with  Databricks  and big data frameworks. Expertise in integrating data science workflows and deploying ML models for real-time and batch processing within a cybersecurity context.
Cloud Proficiency:  Advanced proficiency in AWS, including EC2, S3, Lambda, ELB, and container orchestration (Docker, Kubernetes). Experience in managing large-scale data environments on AWS, optimizing for performance, security, and compliance.
Security Integration:  Proven experience implementing SCAS, SAST, DAST/WAS, and secure DevOps practices within an SDLC framework to ensure data security and compliance in a high-stakes cybersecurity environment.
Data Architecture:  Demonstrated ability to design and implement complex data architectures, including data lakes, data warehouses, and lake house solutions. Emphasis on secure, scalable, and highly available data structures that support ML-driven insights and real-time analytics.
Data Quality & Governance:  Hands-on experience with automated data quality checks, data lineage, and governance standards. Proficiency in Databricks DQM or similar tools to enforce data integrity and compliance across pipelines.
Data Analytics & Visualization:  Proficiency with analytics and visualization tools such as Databricks, Power BI, and Tableau to generate actionable insights for cybersecurity risks, threat patterns, and vulnerability trends. Skilled in translating complex data into accessible visuals and reports for cross-functional teams.
CI/CD and Automation:  Experience building CI/CD pipelines that automate testing, security scans, and deployment processes. Proficiency in deploying ML models and data processing workflows using CI/CD, ensuring consistent quality and streamlined delivery.
Agile Experience:  Deep experience in Agile/Scrum environments, with a thorough understanding of Agile core values and principles, effectively delivering complex projects with agility and cross-functional collaboration.
Preferred Experience
Advanced Data Modeling & Governance:  Expertise in designing data models for cybersecurity data analytics, emphasizing data lineage, federation, governance, and compliance. Experience ensuring security and privacy within data architectures.
Machine Learning & Predictive Analytics:  Experience deploying ML algorithms, predictive models, and anomaly detection frameworks to bolster CASM platforms cybersecurity capabilities.
High-Performance Engineering Culture:  Background in mentoring engineers in data engineering best practices, promoting data science, ML, and analytics integration, and fostering a culture of collaboration and continuous improvement.

Pay: $80-90/hr As a Senior Data Engineer, Education & Qualifications Education: B.S., M.S., or Ph.D. in Computer Science, Data Science, Information Systems, or a related field, or equivalent professional experience. Technical Expertise: 8+ years in data engineering with strong skills in Python, PySpark, SQL, and extensive, hands-on experience with Databricks and big data frameworks. Expertise in integrating data science workflows and deploying ML models for real-time and batch processing within a cybersecurity context. Cloud Proficiency: Advanced proficiency in AWS, including EC2, S3, Lambda, ELB, and container orchestration (Docker, Kubernetes). Experience in managing large-scale data environments on AWS, optimizing for performance, security, and compliance. Security Integration: Proven experience implementing SCAS, SAST, DAST/WAS, and secure DevOps practices within an SDLC framework to ensure data security and compliance in a high-stakes cybersecurity environment. Data Architecture: Demonstrated ability to design and implement complex data architectures, including data lakes, data warehouses, and lake house solutions. Emphasis on secure, scalable, and highly available data structures that support ML-driven insights and real-time analytics. Data Quality & Governance: Hands-on experience with automated data quality checks, data lineage, and governance standards. Proficiency in Databricks DQM or similar tools to enforce data integrity and compliance across pipelines. Data Analytics & Visualization: Proficiency with analytics and visualization tools such as Databricks, Power BI, and Tableau to generate actionable insights for cybersecurity risks, threat patterns, and vulnerability trends. Skilled in translating complex data into accessible visuals and reports for cross-functional teams. CI/CD and Automation: Experience building CI/CD pipelines that automate testing, security scans, and deployment processes. Proficiency in deploying ML models and data processing workflows using CI/CD, ensuring consistent quality and streamlined delivery. Agile Experience: Deep experience in Agile/Scrum environments, with a thorough understanding of Agile core values and principles, effectively delivering complex projects with agility and cross-functional collaboration. Preferred Experience Advanced Data Modeling & Governance: Expertise in designing data models for cybersecurity data analytics, emphasizing data lineage, federation, governance, and compliance. Experience ensuring security and privacy within data architectures. Machine Learning & Predictive Analytics: Experience deploying ML algorithms, predictive models, and anomaly detection frameworks to bolster CASM platforms cybersecurity capabilities. High-Performance Engineering Culture: Background in mentoring engineers in data engineering best practices, promoting data science, ML, and analytics integration, and fostering a culture of collaboration and continuous improvement.

Education & Qualifications
Minimum Qualifications
•  Education:  B.S., M.S., or Ph.D. in Computer Science, Data Science, Information Systems, or a related field, or equivalent professional experience.
•  Technical Expertise:  8+ years in data engineering with strong skills in Python, PySpark, SQL, and extensive, hands-on experience with  Databricks  and big data frameworks. Expertise in integrating data science workflows and deploying ML models for real-time and batch processing within a cybersecurity context.
•  Cloud Proficiency:  Advanced proficiency in AWS, including EC2, S3, Lambda, ELB, and container orchestration (Docker, Kubernetes). Experience in managing large-scale data environments on AWS, optimizing for performance, security, and compliance.
•  Security Integration:  Proven experience implementing SCAS, SAST, DAST/WAS, and secure DevOps practices within an SDLC framework to ensure data security and compliance in a high-stakes cybersecurity environment.
•  Data Architecture:  Demonstrated ability to design and implement complex data architectures, including data lakes, data warehouses, and lake house solutions. Emphasis on secure, scalable, and highly available data structures that support ML-driven insights and real-time analytics.
•  Data Quality & Governance:  Hands-on experience with automated data quality checks, data lineage, and governance standards. Proficiency in Databricks DQM or similar tools to enforce data integrity and compliance across pipelines.
•  Data Analytics & Visualization:  Proficiency with analytics and visualization tools such as Databricks, Power BI, and Tableau to generate actionable insights for cybersecurity risks, threat patterns, and vulnerability trends. Skilled in translating complex data into accessible visuals and reports for cross-functional teams.
•  CI/CD and Automation:  Experience building CI/CD pipelines that automate testing, security scans, and deployment processes. Proficiency in deploying ML models and data processing workflows using CI/CD, ensuring consistent quality and streamlined delivery.
•  Agile Experience:  Deep experience in Agile/Scrum environments, with a thorough understanding of Agile core values and principles, effectively delivering complex projects with agility and cross-functional collaboration.

Preferred Experience
•  Advanced Data Modeling & Governance:  Expertise in designing data models for cybersecurity data analytics, emphasizing data lineage, federation, governance, and compliance. Experience ensuring security and privacy within data architectures.
•  Machine Learning & Predictive Analytics:  Experience deploying ML algorithms, predictive models, and anomaly detection frameworks to bolster CASM platform's cybersecurity capabilities.
•  High-Performance Engineering Culture:  Background in mentoring engineers in data engineering best practices, promoting data science, ML, and analytics integration, and fostering a culture of collaboration and continuous improvement.

Pay: $80-90/hr As a Senior Data Engineer, Education & Qualifications• Education: B.S., M.S., or Ph.D. in Computer Science, Data Science, Information Systems, or a related field, or equivalent professional experience.• Technical Expertise: 8+ years in data engineering with strong skills in Python, PySpark, SQL, and extensive, hands-on experience with Databricks and big data frameworks. Expertise in integrating data science workflows and deploying ML models for real-time and batch processing within a cybersecurity context.• Cloud Proficiency: Advanced proficiency in AWS, including EC2, S3, Lambda, ELB, and container orchestration (Docker, Kubernetes). Experience in managing large-scale data environments on AWS, optimizing for performance, security, and compliance.• Security Integration: Proven experience implementing SCAS, SAST, DAST/WAS, and secure DevOps practices within an SDLC framework to ensure data security and compliance in a high-stakes cybersecurity environment.• Data Architecture: Demonstrated ability to design and implement complex data architectures, including data lakes, data warehouses, and lake house solutions. Emphasis on secure, scalable, and highly available data structures that support ML-driven insights and real-time analytics.• Data Quality & Governance: Hands-on experience with automated data quality checks, data lineage, and governance standards. Proficiency in Databricks DQM or similar tools to enforce data integrity and compliance across pipelines.• Data Analytics & Visualization: Proficiency with analytics and visualization tools such as Databricks, Power BI, and Tableau to generate actionable insights for cybersecurity risks, threat patterns, and vulnerability trends. Skilled in translating complex data into accessible visuals and reports for cross-functional teams.• CI/CD and Automation: Experience building CI/CD pipelines that automate testing, security scans, and deployment processes. Proficiency in deploying ML models and data processing workflows using CI/CD, ensuring consistent quality and streamlined delivery.• Agile Experience: Deep experience in Agile/Scrum environments, with a thorough understanding of Agile core values and principles, effectively delivering complex projects with agility and cross-functional collaboration. Preferred Experience• Advanced Data Modeling & Governance: Expertise in designing data models for cybersecurity data analytics, emphasizing data lineage, federation, governance, and compliance. Experience ensuring security and privacy within data architectures.• Machine Learning & Predictive Analytics: Experience deploying ML algorithms, predictive models, and anomaly detection frameworks to bolster CASM platform’s cybersecurity capabilities.• High-Performance Engineering Culture: Background in mentoring engineers in data engineering best practices, promoting data science, ML, and analytics integration, and fostering a culture of collaboration and continuous improvement.

JOB DESCRIPTION Pay Range: $75-85/hr on W2 Team: Team is Client delivery platform and Cumulus - lots of data exchanges: AWS platform using Redshift as data warehouse - Dynamo of meta-data. They are working on evolving code for digital SAT products and need dynamic Senior Data engineer who will also design, build and write test automation. Profile: Senior Data Engineer with SME in data analytics end AI work within our tech stack; 5+ years of experience. Summary Data Engineer must be a skilled engineer with an unwavering passion for excellence in Software Engineering and drive to transform that passion into consistent excellence in product development. The candidate must be customer-obsessed, have strong analytical and communication skills, enjoy working with native AWS services, and thrive on solving challenging business problems. The candidate will partner with Product Owners/Architects, Developers, and other Software Engineers to help design, define, implement functional features, system level validation and verify features of product development. Qualified candidates will show an aptitude learning and implementation of advanced software engineering practices in Data Analytics, AWS, Microservices environment to solve complex problems with a mission to release high quality software that is resilient and optimally performing. Candidates will be engaged in both data and analytics reports development and quality engineering, testing and automation as a data engineer. Responsibilities Collaborate with and across Agile teams to design, develop, test, implement, and support technical solutions in full-stack development tools and technologies Builds, tests, deploys and maintains production code for complex transactional applications using event driven microservices, node.js, Python languages. Incorporates Cloud technologies on new application development to include micro-services, SQL and AWS services such as lambda, S3, SQS, SNS, DynamoDB, Quicksight, AWS Redshift, Opensearch. Perform unit tests and conduct reviews with other team members to make sure your code is rigorously designed, elegantly coded, and effectively tuned for performance Required Skills •Expert in writing and optimizing complex SQL queries for data extraction, transformation, and reporting (e.g., SELECT, JOIN, GROUP BY, HAVING, WINDOW functions). •Expert integrating Redshift Serverless with AWS services such as S3, Glue, Lambda, Athena, and Kinesis BI tools like Tableau, Quicksight, or Looker for real-time analytics and dashboards •Expert in building serverless ETL pipelines with Lambda to automate data ingestion and transformation into Redshift. •Experience in integrating Redshift Serverless with •Experience of NodeJS and Python •Working knowledge of any AI technologies •Experience in working with AWS services, deploying applications to AWS •Exposure to DevOps tools and automation •Effective communication skills are a must along with a strong customer service orientation, and the ability to clearly discern client needs.

Job description:    Ability to design, write and test software related to messaging systems with a background in message translations. Experience in Agile CI/CD software development deployments. Experience in containerizing packages and deploying to Kubernetes with HELM charts. The software developer will be responsible for development from architecture system design into working software products. C++, and/or Java as well as Python experience required.
Must have skills:

The submitted candidate must have 6 years of work experience in each of the following skills:   Full stack development
K8 (Kubernetes)
Python
JAVA
Tactical Message Translation
Pub/Sub (AMQ and Kafka)
Model Based Development
Agile (JIRA)
CI/CD deployment
HELM Charts
Container Image Creation and Sustainment
Familiarity with Database technology
Important information:    Job title : Data Engineer (Data Modeling)
Must be a U.S. Citizen.
The position is unclassified but the submitted candidate must be able to obtain a clearance.
The position is remote

Data Engineer Lancesoft, Inc. Jobs

Principal Data Engineer Architect
December 8, 2024

Database Engineer Data Engineerdatabase Administrator
December 8, 2024

Senior Data Engineer Casm Platform
December 6, 2024

Data Engineer
December 3, 2024

Data Engineer
December 2, 2024

Senior Data Engineer Casm Platform
December 1, 2024

Data Engineer
December 1, 2024

Senior Data Engineer
December 1, 2024

Data Engineer Data Modeling Remote
November 24, 2024

Quick filters

Top companies

Data Engineer Lancesoft, Inc. Jobs

Principal Data Engineer ArchitectDecember 8, 2024

Database Engineer Data Engineerdatabase AdministratorDecember 8, 2024

Senior Data Engineer Casm PlatformDecember 6, 2024

Data EngineerDecember 3, 2024

Data EngineerDecember 2, 2024

Senior Data Engineer Casm PlatformDecember 1, 2024

Data EngineerDecember 1, 2024

Senior Data EngineerDecember 1, 2024

Data Engineer Data Modeling RemoteNovember 24, 2024