Project Aegis: GDPR-Compliant Data Lakehouse Pipeline

The Project Aegis Data Pipeline is an enterprise-grade, privacy-centric data engineering solution built natively within the Databricks Lakehouse platform. Designed for a e-commerce enterprise, this solution ingests vast amounts of customer interaction, transaction, and behavioral data while enforcing strict General Data Protection Regulation (GDPR) requirements. By leveraging a "Privacy by Design" architecture, the system automates the classification of Personally Identifiable Information (PII), applies dynamic data masking, and orchestrates automated workflows for Data Subject Access Requests (DSAR) and the "Right to be Forgotten" (RTBF). The pipeline seamlessly integrates Databricks Unity Catalog, Delta Lake, and PySpark to deliver analytics-ready data to business users without compromising regulatory compliance or data security.

Industry: Retail | E-Commerce

Technologies: PySpark | Delta Lake | Apache Kafka | SQL

Tools/Libraries: HTML | Databricks Lakehouse Platform | Unity Catalog | Databricks Workflows | Azure Key Vault

Igniting Powerful INSIGHTS

The Challenge

The Challenge The enterprise faced significant regulatory and technical hurdles as it scaled its operations across the EU: Manual & Slow Data Subject Requests: Responding to DSARs (Article 15) and RTBF requests (Article 17) took weeks. Data was scattered across operational SQL databases, S3 blob storage, and historical parquet files, making it nearly impossible to locate and delete a specific user's data comprehensively. Immutable Data Lake Limitations: The existing data lake architecture was append-only. While great for historical tracking, it violated the GDPR mandate for physical data deletion within 30 days of a request. PII Exposure Risk: Data analysts had broad access to raw data zones, exposing sensitive customer attributes (emails, IP addresses, phone numbers) unnecessarily, risking non-compliance with the GDPR data minimization principle. Lack of Auditability: The enterprise lacked a centralized, tamper-proof audit log tracking who accessed PII and what modifications were made to user records.

Goals / Objectives

The primary objectives of the solution were to: 1. Automate the fulfillment of DSAR and RTBF requests, reducing turnaround time from weeks to under 24 hours. 2. Implement strict data minimization and access controls using dynamic data masking and row-level security. 3. Ensure physical deletion of user data across all layers of the Medallion architecture (Bronze, Silver, Gold) while maintaining pipeline idempotency. 4. Establish a centralized governance framework with automated PII discovery and immutable audit logging. 5. Provide a secure analytics environment where data science teams can build models without accessing raw PII.

Solution / Approach

The solution was architected as a Privacy-by-Design Medallion Framework utilizing Databricks Delta Lake and Unity Catalog. Instead of treating compliance as an afterthought, governance was embedded directly into the ingestion and transformation logic. The architecture follows a multi-hop approach with strict privacy boundaries: Bronze Layer (Raw & Quarantined): Ingests raw data from upstream Kafka streams and Azure SQL. This layer is strictly isolated, accessible only by automated service principals. Data is stored using Delta Lake to enable transactional deletes. Silver Layer (Pseudonymized): PII columns are tokenized using SHA-256 hashing with a salt. Direct identifiers (e.g., email) are replaced with anonymous tokens. A secure, isolated lookup table maps tokens to original PII, accessible only by the Data Privacy Officer (DPO). Gold Layer (Aggregated & Minimized): Contains business-ready tables where data is aggregated. No direct PII exists here; only tokens and necessary behavioral metrics are retained. To handle data deletion, a custom RTBF Orchestrator Job was built. This job dynamically generates PySpark code to locate user records across all layers, executes Delta `DELETE` operations, and physically purges the underlying Parquet files using `VACUUM`. Unity Catalog acts as the central brain, automating PII tagging and enforcing Dynamic Data Masking rules at query runtime.

Process / Workflow

The end-to-end workflow operates through two parallel streams: Data Ingestion/Transformation and Compliance Orchestration. 1. Source Ingestion & PII Tagging: Relational & Streaming Sources: Customer profiles and transaction logs are ingested via Databricks Autoloader and JDBC. Automated Classification: Unity Catalog automatically scans incoming data, utilizing a custom metadata table to identify columns like `user_email`, `phone_number`, and `ip_address` as PII. 2. Transformation & Pseudonymization: A PySpark notebook reads the Bronze layer. Using a UDF (User Defined Function) backed by Azure Key Vault, it hashes PII columns before writing to the Silver layer. The Gold layer aggregates this data for BI consumption, utilizing the hashed tokens as join keys, ensuring business logic remains intact without exposing identities. 3. Data Subject Access Request (DSAR) Workflow: A DPO submits a user ID via a REST API trigger. A Databricks Job queries all Delta tables across Bronze, Silver, and Gold for that specific ID. The results are compiled into a secure JSON/CSV file, written to an isolated Databricks Volume, and a notification is sent to the DPO for user delivery. 4. Right to be Forgotten (RTBF) Execution: Upon receiving a deletion request, the RTBF Orchestrator Job initiates. Logical Deletion: It executes targeted `DELETE FROM` statements on Delta tables in Bronze and Silver where the user ID matches. Physical Deletion: The job runs `OPTIMIZE` and then `VACUUM` with a 0-hour retention threshold on the affected tables, physically purging the old Parquet files containing the user's data from cloud storage. 5. Automated Audit Logging: Every step—from API request to Delta table modification—is logged into an immutable Delta table governed by Unity Catalog, providing undeniable proof of compliance for auditors.

Key Features

Automated RTBF Execution: Programmatic logical and physical deletion of user records across multi-layered data pipelines, fulfilling Article 17 mandates within minutes.
Dynamic Data Masking: Unity Catalog policies automatically mask PII (e.g., masking all but the last 4 digits of a phone number) for specific user groups, while granting full access to authorized compliance officers.
Pseudonymization Engine: Secure, salted hashing of direct identifiers allows data scientists to track user journeys across sessions without accessing real identities.
DSAR Automation API: An API-integrated Databricks job that compiles a user's entire data footprint into a single report on demand.
Immutable Audit Trail: A tamper-proof logging mechanism tracking all data access, modifications, and deletion events.

Technical Highlights

Delta Lake Deletion Vectors: Enabled Delta Lake Deletion Vectors to optimize the performance of `DELETE` operations during RTBF processing, avoiding costly file rewrites and ensuring concurrent read operations by analysts are not interrupted.
Physical Purging via VACUUM: Configured safe, automated `VACUUM` operations bypassing standard 7-day retention checks specifically for RTBF compliance, paired with cloud-level lifecycle policies to ensure physical destruction of underlying blobs.
Unity Catalog Governance: Implemented column-level masking tags (e.g., `TAG: PII = TRUE`). PySpark UDFs dynamically applied masking rules based on the executing user's Unity Catalog group membership.
Secure Tokenization UDFs: Developed PySpark UDFs that securely communicate with Azure Key Vault to retrieve salting keys, ensuring that the re-identification of pseudonymized data is impossible within the analytics environment.
Event-Driven Orchestration: Utilized Databricks Workflows triggered by AWS Lambda/Azure Functions via the Databricks REST API, enabling real-time execution of DSAR and RTBF requests directly from the company's privacy portal.

Benefits / Impact

Exponential SLA Improvement: Reduced the turnaround time for Data Subject Access Requests and Deletion requests from an average of 18 days to under 4 hours.
Zero Compliance Violations: Achieved a 100% compliance rate during external GDPR audits, backed by the immutable audit trail and verified physical deletion logs.
Safe Data Democratization: Increased data accessibility for data scientists and analysts by 60%. Teams can now query Silver and Gold tables freely, as dynamic masking and pseudonymization guarantee no accidental PII exposure.
Reduced Legal Risk: By automating compliance, the enterprise mitigated the risk of severe GDPR fines (up to 4% of global revenue) and significantly reduced the manual operational overhead of the legal and compliance teams.
Optimized Storage: The use of Deletion Vectors and targeted `VACUUM` operations ensured that compliance deletions did not bloat cloud storage costs or degrade query performance.

Conclusion

Project Aegis represents a benchmark for modern, privacy-centric data engineering. By deeply integrating GDPR requirements directly into the Databricks Lakehouse architecture—rather than bolting them on as an afterthought—the enterprise achieved a secure, scalable, and highly automated data ecosystem. The use of Delta Lake's advanced features, combined with Unity Catalog's granular governance, allowed the organization to balance rigorous regulatory compliance with the need for agile, data-driven innovation. Moving forward, this framework can be easily adapted to comply with other global privacy regulations (such as CCPA or HIPAA). Its future potential includes integrating NLP-based automated PII discovery models to dynamically tag unstructured data (like customer support transcripts) and expanding the RTBF orchestrator to handle cross-cloud data lakehouse environments seamlessly.