Data Engineer / Data Analyst
Nov 2024 - PresentSolvento Philippines, Inc. - Makati, Metro Manila, Philippines
- Developed and optimized data profiling scripts using Python and SQL improving data quality assessment speed across multiple sources
- Developed and maintained a data ingestion pipeline for transforming and validating 200MB+ CSV files (2M+ rows), using Python for orchestration and small file handling, while leveraging Rust for performance critical components to improve execution speed and memory efficiency.
- Transformed and prepared data for reporting, and built automated reporting solutions with Python and SQL, significantly reducing manual processes and improving accuracy.
- Executed end-to-end migration of on-premises Microsoft SQL Server historical databases to an Azure Virtual Machine, ensuring data integrity, system stability, and minimal downtime, while architecting and managing backup and recovery solutions.
- Designed and implemented an ETL pipeline using .NET Framework 4 to extract raw transactional data from production SQL Server databases and load it into a Bronze layer within a Windows VM environment, optimizing large-volume data ingestion through SqlBulkCopy-based batch processing.
- Set up and administered Power BI On-Premises Data Gateways, configuring data source connections and credentials to support scheduled refresh and secure data access from MSSQL databases.
- Assisted in building Power BI dashboards (Finance, Commercial, and CEO dashboards) by developing and optimizing DAX measures, collaborating on visualization design, and working with gold-layer data to support reporting and analytics.
- Analyzed and optimized production MSSQL stored procedures, achieving up to 80% performance improvement through query tuning, indexing strategies, and execution plan optimization.
- Refactored ETL Python pipelines by implementing modular, reusable functions and improving code architecture, increasing maintainability and scalability in preparation for Microsoft Fabric migration