Lead the development and optimization of data systems, migrating legacy systems to Google BigQuery, and building robust ETL pipelines for real-time data integration.
About Brillio:
Brillio is one of the fastest growing digital technology service providers and a partner of choice for many Fortune 1000 companies seeking to turn disruption into a competitive advantage through innovative digital adoption. Brillio, renowned for its world-class professionals, referred to as "Brillians", distinguishes itself through their capacity to seamlessly integrate cutting-edge digital and design thinking skills with an unwavering dedication to client satisfaction.
Brillio takes pride in its status as an employer of choice, consistently attracting the most exceptional and talented individuals due to its unwavering emphasis on contemporary, groundbreaking technologies, and exclusive digital projects. Brillio's relentless commitment to providing an exceptional experience to its Brillians and nurturing their full potential consistently garners them the Great Place to Work® certification year after year.
Lead Data Engineer
Primary Skills
- IICS, Alation, Data Modelling Fundamentals, Data Warehousing, ETL Fundamentals, Modern Data Platform Fundamentals, PLSQL, T-SQL, Stored Procedures, Python, SQL, SQL (Basic + Advanced), Talend
Job requirements
- Job Title: Lead Data Engineer
- We are seeking a highly skilled and experienced Senior Data Engineer to lead the development, migration, and optimization of data systems across our enterprise infrastructure.
- This role will play a critical part in migrating legacy data systems to Google BigQuery, designing robust ETL/data pipelines, and enabling real-time IoT data integration.
- The ideal candidate will have a strong background in cloud-based data warehousing, particularly in Google Cloud Platform (GCP), and deep expertise in Python, SQL, Spark, and data engineering best practices.
- 1. Data Migration & Cloud Modernization Analyze legacy on-premises and cloud data warehouse systems (e.g., SQL Server). Lead migration of large-scale datasets to Google BigQuery. Design and execute data migration strategies ensuring data quality, consistency, and performance.
- 2. Data Integration & Streaming Integrate data from diverse sources including APIs, relational databases, and unstructured data. Implement real-time streaming pipelines for IoT data ingestion and processing in large-scale environments.
- 3. ETL / Data Pipeline Development Evaluate and modernize legacy SSIS packages to cloud-native ETL pipelines. Design and develop scalable ETL workflows using Apache Airflow, Python, Spark, and other GCP-native tools. Ensure data is processed, transformed, and loaded reliably into BigQuery for downstream analytics.
- 4. Programming & Query Optimization Write complex SQL queries, stored procedures, and scheduled queries in BigQuery. Build reusable, modular data transformation scripts using Python, Java, Spark, and SQL. Optimize query performance and manage cost efficiency in a cloud data warehouse environment.
- 5+ years of experience in Data Engineering, including at least 2+ years working with Google Cloud Platform (GCP) and BigQuery.
- Strong experience with data migration from on-prem to cloud platforms. Proficiency in building ETL pipelines with Apache Airflow, Python, and Spark.
- Hands-on experience with streaming data ingestion (especially IoT data).
- Strong SQL development skills; experience with performance tuning in BigQuery.
- Solid understanding of cloud architecture and data warehouse design.
- Experience with version control systems (e.g., Git) and CI/CD for data pipelines.
- GCP Professional Data Engineer certification.
- Experience with tools like dbt, Terraform, or Kafka.
- Background in analytics, machine learning pipelines, or DevOps for data.
- #LI-SR1
Job Overview:
Key Responsibilities:
Required Skills & Experience:
Preferred Qualifications:
Know more about:
DAI: https://www.brillio.com/services-data-analytics/
Know what it’s like to work and grow at Brillio: https://www.brillio.com/join-us/
Equal Employment Opportunity Declaration
Brillio is an equal opportunity employer to all, regardless of age, ancestry, colour, disability (mental and physical), exercising the right to family care and medical leave, gender, gender expression, gender identity, genetic information, marital status, medical condition, military or veteran status, national origin, political affiliation, race, religious creed, sex (includes pregnancy, childbirth, breastfeeding, and related medical conditions), and sexual orientation.
Know what it’s like to work and grow at Brillio: Click here
Similar Jobs
Financial Services
Manage accounting operations, including reconciliations, journal entries, accounts receivable, accounts payable, billing analysis, revenue recognition, financial statements, budgeting, accruals, tax schedules, audits, and process improvements. The role ensures GAAP compliance, supports financial reporting and forecasting, analyzes expenditures and billing transactions, and communicates financial information across the organization.
Top Skills:
GaapIntacctExcelNetSuiteQuickbooks
Financial Services
Despite the conflicting description, the posted title is Senior Data Scientist. The body primarily describes an accounting role responsible for reconciliations, journal entries, billing, revenue recognition, financial statements, budgeting, audits, tax schedules, GAAP compliance, and process improvements. The position requires accounting expertise, financial analysis, Excel proficiency, and experience with accounting systems.
Top Skills:
IntacctExcelNetSuiteQuickbooks
Financial Services
Manage accounting operations, including reconciliations, journal entries, accounts receivable, accounts payable, billing analysis, revenue recognition, and general ledger activities. Prepare financial statements, budgets, variance analyses, tax schedules, and audit support. Ensure GAAP compliance, analyze expenditures, and improve accounting processes while collaborating with departments and management.
Top Skills:
GaapIntacctExcelNetSuiteQuickbooksRevenue Recognition
What you need to know about the San Francisco Tech Scene
San Francisco and the surrounding Bay Area attracts more startup funding than any other region in the world. Home to Stanford University and UC Berkeley, leading VC firms and several of the world’s most valuable companies, the Bay Area is the place to go for anyone looking to make it big in the tech industry. That said, San Francisco has a lot to offer beyond technology thanks to a thriving art and music scene, excellent food and a short drive to several of the country’s most beautiful recreational areas.
Key Facts About San Francisco Tech
- Number of Tech Workers: 365,500; 13.9% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Google, Apple, Salesforce, Meta
- Key Industries: Artificial intelligence, cloud computing, fintech, consumer technology, software
- Funding Landscape: $50.5 billion in venture capital funding in 2024 (Pitchbook)
- Notable Investors: Sequoia Capital, Andreessen Horowitz, Bessemer Venture Partners, Greylock Partners, Khosla Ventures, Kleiner Perkins
- Research Centers and Universities: Stanford University; University of California, Berkeley; University of San Francisco; Santa Clara University; Ames Research Center; Center for AI Safety; California Institute for Regenerative Medicine
