The Project Aegis Data Pipeline is an enterprise-grade, privacy-centric data engineering solution built natively within the Databricks Lakehouse platform. Designed for a e-commerce enterprise, this solution ingests vast amounts of customer interaction, transaction, and behavioral data while enforcing strict General Data Protection Regulation (GDPR) requirements. By leveraging a "Privacy by Design" architecture, the system automates the classification of Personally Identifiable Information (PII), applies dynamic data masking, and orchestrates automated workflows for Data Subject Access Requests (DSAR) and the "Right to be Forgotten" (RTBF). The pipeline seamlessly integrates Databricks Unity Catalog, Delta Lake, and PySpark to deliver analytics-ready data to business users without compromising regulatory compliance or data security.
The Challenge The enterprise faced significant regulatory and technical hurdles as it scaled its operations across the EU: Manual & Slow Data Subject Requests: Responding to DSARs (Article 15) and RTBF requests (Article 17) took weeks. Data was scattered across operational SQL databases, S3 blob storage, and historical parquet files, making it nearly impossible to locate and delete a specific user's data comprehensively. Immutable Data Lake Limitations: The existing data lake architecture was append-only. While great for historical tracking, it violated the GDPR mandate for physical data deletion within 30 days of a request. PII Exposure Risk: Data analysts had broad access to raw data zones, exposing sensitive customer attributes (emails, IP addresses, phone numbers) unnecessarily, risking non-compliance with the GDPR data minimization principle. Lack of Auditability: The enterprise lacked a centralized, tamper-proof audit log tracking who accessed PII and what modifications were made to user records.
The primary objectives of the solution were to: 1. Automate the fulfillment of DSAR and RTBF requests, reducing turnaround time from weeks to under 24 hours. 2. Implement strict data minimization and access controls using dynamic data masking and row-level security. 3. Ensure physical deletion of user data across all layers of the Medallion architecture (Bronze, Silver, Gold) while maintaining pipeline idempotency. 4. Establish a centralized governance framework with automated PII discovery and immutable audit logging. 5. Provide a secure analytics environment where data science teams can build models without accessing raw PII.
The solution was architected as a Privacy-by-Design Medallion Framework utilizing Databricks Delta Lake and Unity Catalog. Instead of treating compliance as an afterthought, governance was embedded directly into the ingestion and transformation logic. The architecture follows a multi-hop approach with strict privacy boundaries: Bronze Layer (Raw & Quarantined): Ingests raw data from upstream Kafka streams and Azure SQL. This layer is strictly isolated, accessible only by automated service principals. Data is stored using Delta Lake to enable transactional deletes. Silver Layer (Pseudonymized): PII columns are tokenized using SHA-256 hashing with a salt. Direct identifiers (e.g., email) are replaced with anonymous tokens. A secure, isolated lookup table maps tokens to original PII, accessible only by the Data Privacy Officer (DPO). Gold Layer (Aggregated & Minimized): Contains business-ready tables where data is aggregated. No direct PII exists here; only tokens and necessary behavioral metrics are retained. To handle data deletion, a custom RTBF Orchestrator Job was built. This job dynamically generates PySpark code to locate user records across all layers, executes Delta `DELETE` operations, and physically purges the underlying Parquet files using `VACUUM`. Unity Catalog acts as the central brain, automating PII tagging and enforcing Dynamic Data Masking rules at query runtime.
The end-to-end workflow operates through two parallel streams: Data Ingestion/Transformation and Compliance Orchestration. 1. Source Ingestion & PII Tagging: Relational & Streaming Sources: Customer profiles and transaction logs are ingested via Databricks Autoloader and JDBC. Automated Classification: Unity Catalog automatically scans incoming data, utilizing a custom metadata table to identify columns like `user_email`, `phone_number`, and `ip_address` as PII. 2. Transformation & Pseudonymization: A PySpark notebook reads the Bronze layer. Using a UDF (User Defined Function) backed by Azure Key Vault, it hashes PII columns before writing to the Silver layer. The Gold layer aggregates this data for BI consumption, utilizing the hashed tokens as join keys, ensuring business logic remains intact without exposing identities. 3. Data Subject Access Request (DSAR) Workflow: A DPO submits a user ID via a REST API trigger. A Databricks Job queries all Delta tables across Bronze, Silver, and Gold for that specific ID. The results are compiled into a secure JSON/CSV file, written to an isolated Databricks Volume, and a notification is sent to the DPO for user delivery. 4. Right to be Forgotten (RTBF) Execution: Upon receiving a deletion request, the RTBF Orchestrator Job initiates. Logical Deletion: It executes targeted `DELETE FROM` statements on Delta tables in Bronze and Silver where the user ID matches. Physical Deletion: The job runs `OPTIMIZE` and then `VACUUM` with a 0-hour retention threshold on the affected tables, physically purging the old Parquet files containing the user's data from cloud storage. 5. Automated Audit Logging: Every step—from API request to Delta table modification—is logged into an immutable Delta table governed by Unity Catalog, providing undeniable proof of compliance for auditors.
Enabled Delta Lake Deletion Vectors to optimize the performance of `DELETE` operations during RTBF processing, avoiding costly file rewrites and ensuring concurrent read operations by analysts are not interrupted.
Configured safe, automated `VACUUM` operations bypassing standard 7-day retention checks specifically for RTBF compliance, paired with cloud-level lifecycle policies to ensure physical destruction of underlying blobs.
Implemented column-level masking tags (e.g., `TAG: PII = TRUE`). PySpark UDFs dynamically applied masking rules based on the executing user's Unity Catalog group membership.
Developed PySpark UDFs that securely communicate with Azure Key Vault to retrieve salting keys, ensuring that the re-identification of pseudonymized data is impossible within the analytics environment.
Utilized Databricks Workflows triggered by AWS Lambda/Azure Functions via the Databricks REST API, enabling real-time execution of DSAR and RTBF requests directly from the company's privacy portal.
Project Aegis represents a benchmark for modern, privacy-centric data engineering. By deeply integrating GDPR requirements directly into the Databricks Lakehouse architecture—rather than bolting them on as an afterthought—the enterprise achieved a secure, scalable, and highly automated data ecosystem. The use of Delta Lake's advanced features, combined with Unity Catalog's granular governance, allowed the organization to balance rigorous regulatory compliance with the need for agile, data-driven innovation. Moving forward, this framework can be easily adapted to comply with other global privacy regulations (such as CCPA or HIPAA). Its future potential includes integrating NLP-based automated PII discovery models to dynamically tag unstructured data (like customer support transcripts) and expanding the RTBF orchestrator to handle cross-cloud data lakehouse environments seamlessly.
