About me

Hello! I’m an Azure Data Engineer with over 8 years of experience in Big Data Engineering and Generative AI. I transform raw data into actionable insights and deliver production-ready GenAI applications that solve real-world challenges.

My expertise spans end-to-end data pipelines and scalable systems on Microsoft Azure. From data lakes to AI-driven applications, I help organizations unlock data’s potential to drive strategic goals.

I specialize in integrating AI solutions into production, leveraging tools like Azure Databricks, PySpark, LangChain, and more. Check out my Portfolio tab for projects in Data Engineering, Machine Learning, and GenAI.

Programming Language

  • Pyspark
    100%
  • SQL
    100%
  • Python
    90%

Libraries

Daniel lewis
quote icon

Daniel lewis

Certifications

Resume


Experience

  1. Data Engineer

    Fractal Analytics | Jan 2023 — Present

    - Developed scalable metadata pipelines using Databricks, Azure tools, and SQL, enhancing data scalability for a retail client.
    - Contributed to ODS project, replicating 50+ transactional tables to reduce system load and enable faster scalability.
    - Built in-store Virtual Assistant with Databricks and Azure OpenAI, improving customer support and ticket classification.
    - Migrated data from Azure Synapse to Microsoft Fabric OneLake, ensuring integrity and optimizing storage architecture.
    - Monitored and optimized pipelines, achieving 99.9% uptime and supporting rapid scaling for high-volume data operations.
    - Collaborated with stakeholders to design scalable solutions, delivering measurable outcomes aligned with growth goals.


  2. Data Engineer

    Accenture AI | Sep 2021 — Dec 2022

    - Designed and deployed robust ETL pipelines for efficient data staging, transformation, and loading into Azure Data Warehouse, prioritizing performance and scalability.
    - Utilized Azure Data Factory to streamline workflow orchestration and monitor pipeline operations, enhancing data integration processes.
    - Applied Azure Databricks and PySpark for advanced data cleansing, transformation, and processing, achieving significant improvements in data quality and processing speed.
    - Authored and optimized SQL scripts for seamless data modeling, extraction, and management tasks.
    - Performed detailed data profiling, validation, and quality assurance to uphold data accuracy and consistency across datasets.


  3. Data Analyst

    Quest Global | Jun 2019 - Sep 2021

    - Conducted comprehensive analysis of diverse datasets including Oil Production, Air Emission, and Electrical data, employing statistical techniques and data visualization tools.
    - Developed and maintained Tableau dashboards, reports, and visualizations, presenting complex data in a concise and compelling manner to facilitate data-driven decision-making.
    - Conducted exploratory data analysis (EDA) to uncover patterns, trends, and correlations in the data, providing actionable insights to support business strategies and process improvements.


  4. Software Engineer

    Asenton Technologies | Feb 2016 - Feb 2018

    - Developed WooCommerce plugins to implement features like infinite scrolling, reward points, cart recovery, and subscription management.
    - Created responsive, visually appealing product landing pages using HTML and CSS.
    - Analyzed digital marketing metrics to optimize campaigns and drive engagement through strategic email initiatives.
    - Fixed front-end issues on customer websites to enhance functionality and user experience.


Education

  1. PG DIPLOMA (Thermal Power Plant Engineering)

    National Power Training Institute | 2018 - 2019

    - Focused on power plant operation and maintenance.


  2. B.E. (Electrical and Electronics Engineering)

    Anna University | 2011 - 2015

    - Core Electrical and Electronics concepts.


Contact

Contact Form