Willis Towers Watson (WTW) is a multinational firm that gives a variety of companies in business insurance coverage brokerage, threat administration, worker advantages, and actuarial evaluation—serving 91% of Fortune World 500 corporations. WTW’s Work and Rewards division delivers data-driven insights, know-how options, and companies to help purchasers’ hiring and retention choices by way of beneficial market-based information.
Our core enterprise depends upon sturdy information transformation and governance capabilities. At its coronary heart is our proprietary calculation engine, which powers our wage survey reviews. Just lately, we started a strategic migration from SQL Server on Azure VMs to Azure Databricks, unlocking vital enterprise benefits. This shift has accelerated report technology by 10x, lowered our information environments by 50%, and minimize storage prices to one-third of our earlier SQL Server bills.
Databricks Unity Catalog has been a key enabler of this transformation, redefining our strategy to information governance and administration. On this weblog, we’ll share the challenges we encountered, how Databricks and Unity Catalog helped us overcome them, and the influence this transformation has had on our enterprise.
Challenges: Scaling, governance, and efficiency Bottlenecks
Earlier than implementing Databricks and Unity Catalog, we confronted a number of technical and organizational challenges stemming from limitations in our current tech stack.
Restricted scalability driving prices, gradual ETL, and delayed time to market: Our report calculation enterprise is seasonal, with distinct peak intervals for information acquisition and report technology. Nonetheless, our legacy database server lacked scalability, forcing us to provision massive database situations year-round—leading to vital idle time and pointless prices. Scaling internet and app servers throughout peak intervals required customized logic in our software code. WTW’s largest reviews usually took between 10 to 36 hours to generate, particularly for our most crucial surveys. These lengthy runtimes created friction in buyer relationships and added operational prices by slowing down different processes.
Rigid schema design in legacy SQL Server implementation limiting agility: Our use of inflexible relational information fashions made it troublesome to adapt to evolving information necessities. This inflexibility led to excessive improvement and upkeep prices.
Information and setting duplication complicating governance: We maintained information throughout greater than 5 environments to satisfy processing and regional compliance wants, incurring over $300K yearly. To make sure quick information retrieval, we relied on roughly 70 TB of P40 storage disks in SQL Server on Azure VMs—including to our infrastructure bills.
Compliance challenges in managing regional information: As a world group, WTW should adhere to regional information privateness and compliance laws. As a consequence of limitations in our prior system, we selected to not accumulate personally identifiable info (PII) to keep away from elevated complexity. Moreover, all information was saved in a single area, limiting our capacity to satisfy localized compliance necessities.
Inadequate information lineage and auditing: We lacked automated instruments to trace information lineage and audit adjustments—each vital for troubleshooting and understanding the downstream influence of information modifications.
Enhancements with Databricks Information Intelligence Platform and Unity Catalog
In our next-generation system constructed on Databricks and Unity Catalog, we’re already realizing a number of advantages that allow our enterprise to scale extra successfully.
10x quicker report technology unlocks enterprise development: Reviews that beforehand took 10 hours to generate now full in below 50 minutes—a 10x enchancment in calculation time. Most reviews are seeing 5x to 20x efficiency beneficial properties. This acceleration is particularly beneficial for our largest reviews, which used to take over a day to course of. Sooner report technology decreases time-to-market, unlocks upsell alternatives by delivering insights to purchasers extra rapidly and often, and frees up our workforce to deal with new consumer engagements.
30% value financial savings by way of Unity Catalog’s environment friendly information administration: By centralizing information entry with Unity Catalog, we have lowered storage and infrastructure prices by 30%. Beforehand, our system relied on six separate environments to satisfy compliance and operational wants. Right now, a single Databricks workspace can seamlessly entry a number of catalogs throughout areas, eliminating duplication and simplifying governance. As well as, storing information in Unity Catalog managed tables—with built-in compression and efficiency optimizations—has minimize storage prices to one-third of our earlier SQL Server bills. These managed tables additionally streamline operations by mechanically optimizing desk layouts primarily based on question patterns.
Improved compliance and information residency capabilities: We are able to now meet privateness necessities from the European Union, Germany, China, California, and different jurisdictions utilizing Unity Catalog’s versatile structure, paired with Databricks’ world availability throughout Azure areas. Storing information nearer to the place it’s accessed has the potential to scale back cloud egress prices and enhance report efficiency. Unity Catalog’s fine-grained entry controls and integration with Entra ID teams additionally give us higher management over PII—benefiting each our compliance posture and our prospects.
Higher auditability and lineage elevated developer productiveness by 33%: Unity Catalog’s lineage options, together with Databricks’ orchestration instruments and AI capabilities, allow higher exploratory information evaluation and deeper understanding of our datasets—particularly as new reviews are developed. Based mostly on our estimates, reviews that beforehand took as much as three weeks to develop now take two weeks or much less.
Databricks coaching accelerated workforce ramp-up: The supply of each free and paid Databricks coaching applications enabled groups with no prior expertise in Databricks or Python to develop into productive inside 2–3 months. By the top of eight months, most builders reached roughly 75% proficiency.
Trying ahead
The migration to Databricks and Unity Catalog has been a game-changer for the Work & Rewards division at Willis Towers Watson. We’ve streamlined information governance, improved compliance, lowered prices, and dramatically accelerated report technology.
Trying forward, we plan to harness Databricks’ AI-powered capabilities—similar to AI/BI Genie and automatic ML options—to construct new data-driven merchandise. With a modernized information infrastructure in place, we’re well-positioned to drive innovation, improve buyer experiences, and unlock new income alternatives.
