How Octus achieved 85% infrastructure price discount with zero downtime migration to Amazon OpenSearch Service

As knowledge volumes proceed to develop exponentially, there may be growing strain to optimize search infrastructure prices whereas sustaining the excessive efficiency and reliability that mission-critical workloads demand. Many firms discover themselves managing advanced, costly search methods that require vital operational overhead and restrict their means to scale effectively. The problem turns into much more acute when organizations have to migrate between search methods, a course of that historically includes substantial downtime, advanced knowledge synchronization, and vital impression on enterprise operations. Enterprise functions can not afford service interruptions that might impression buyer experiences, enterprise intelligence, or operational continuity. Migration methods have to ship price optimization and operational enhancements whereas sustaining zero downtime and facilitating full knowledge integrity all through the transition course of.

Based in 2013, Octus, previously Reorg, is the important credit score intelligence and knowledge supplier for the world’s main purchase facet companies, funding banks, regulation companies and advisory companies. By surrounding unparalleled human experience with confirmed expertise, knowledge and AI instruments, Octus unlocks highly effective truths that gas decisive motion throughout monetary industries.

This put up highlights how Octus migrated its Elasticsearch workloads working on Elastic Cloud to Amazon OpenSearch Service. The journey traces Octus’s shift from managing a number of methods to adopting a cost-efficient answer powered by OpenSearch Service. Alongside the best way, we share the structure selections and implementation methods that made the migration profitable. The result’s uninterrupted service availability all through migration, with improved efficiency and higher price effectivity.

Strategic necessities

We recognized a number of necessities that made Amazon OpenSearch Service the proper alternative for his or her migration:

• Price effectivity: The OpenSearch Service pricing mannequin enabled us to optimize cloud spend with out compromising efficiency.
• Responsive assist: AWS supplied reliable, high-quality assist to speed up concern decision and instill confidence.
• Constant reliability: OpenSearch Service gives an SLA as much as 99.99% providing the reliability required for Octus’s mission-critical workloads.
• Seamless migration with no question downtime: Migration Assistant for Amazon OpenSearch Service supplied Octus with a migration path whereas sustaining uninterrupted question availability in the course of the migration, facilitating enterprise continuity.
• Operational simplification: Consolidating onto AWS decreased infrastructure complexity whereas sustaining excessive safety requirements.

Resolution overview

The Migration Assistant for Amazon OpenSearch Service gives a collection of instruments to assist in Elasticsearch to OpenSearch Service migrations. Octus use the next capabilities for his or her migration:

• Metadata migration: The instrument enabled Octus emigrate dozens of indices with numerous mappings and settings. When a backward incompatibility was recognized with timestamp metadata, a customized JavaScript transformation, built-in immediately into the Migration Assistant tooling, was utilized to mechanically regulate the mappings throughout the indices and facilitate compatibility.
• Historic knowledge migration: Octus used Reindex-from-Snapshot emigrate the historic paperwork from a point-in-time snapshot of the supply cluster, scaling this course of with out impacting the supply cluster because the snapshot was saved in Amazon Easy Storage Service (Amazon S3). Reindex-from-Snapshot additionally enabled Octus to regulate the sharding scheme throughout migration, serving to to optimize cluster efficiency on the goal.
• Stay Site visitors Replay: As soon as backfill was full, Octus used Migration Assistant’s Site visitors Replayer to ship the captured reside visitors (from the Site visitors Seize Proxy) to the goal cluster with required request transformations for OpenSearch Service compatibility, ensuing within the goal cluster containing the paperwork from the supply cluster with updates being carried out in actual time.

The next diagram illustrates the implementation structure diagram for this migration.

Determine 1 – Migration Assistant structure with migration steps

For extra details about the Migration Assistant for Amazon OpenSearch Service, go to the AWS Options dwelling web page.

Every node within the diagram correlates to the next steps within the migration course of:

1. Consumer visitors is directed to the present cluster.
2. An Software Load Balancer with seize proxies relays visitors to a supply whereas replicating knowledge to Amazon Managed Streaming for Apache Kafka (Amazon MSK).
3. Utilizing the migration console, a point-in-time snapshot is taken. As soon as the snapshot completes, the Metadata Migration Instrument is used to ascertain indexes, templates, element templates, and aliases on the goal cluster. With steady visitors seize in place, Reindex-from-Snapshot, migrates knowledge from the supply.
4. As soon as Reindex-from-Snapshot is full, captured visitors is replayed from Amazon Managed Streaming for Apache Kafka (Amazon MSK) to the goal cluster by Site visitors Replayer.
5. Efficiency and conduct of visitors despatched to the supply and goal clusters are in contrast by reviewing logs and metrics.
6. After confirming that the goal cluster’s performance meets expectations, purchasers are redirected to the brand new goal.

Full migration and optimization journey

Octus’s migration from Elastic Cloud to Amazon OpenSearch Service encompassed each the core migration effort and subsequent optimization phases. The aim was to efficiently migrate the search infrastructure, functions, and knowledge from Elastic Cloud to a brand new OpenSearch Service area with minimal disruption, whereas constantly optimizing efficiency and prices based mostly on real-world utilization knowledge.

Octus used their in-house customized infrastructure frameworks (their inner tooling for infrastructure automation) to construct, deploy and monitor the goal OpenSearch Service 1.3 area, establishing a stable basis for the migration. This strategy used acquainted inner processes whereas shifting to the absolutely managed AWS service. Check with AWS documentation to implement safety finest practices when utilizing OpenSearch Service.

Pre-migration optimization

Previous to initiating the migration, Octus carried out optimization actions on the supply Elasticsearch cluster to streamline the migration course of. This included eradicating unused indexes that had gathered over time and eradicating giant paperwork that may unnecessarily lengthen migration period and improve storage switch prices. These preparatory steps considerably decreased the info quantity requiring migration and minimized the general migration complexity, enabling extra environment friendly use of the Migration Assistant instruments.

Technical constraints and model concerns

The migration concerned particular model compatibility challenges that influenced the technical strategy. The supply Elasticsearch cluster was working model 7.17, and the Python consumer functions have been additionally constrained to Elasticsearch 7.17 compatibility. To assist the transition, the staff used Reindex-from-Snapshot, which allows cross-system migrations by reindexing knowledge from present snapshots into a brand new OpenSearch Service cluster. RFS additionally rewrites indices created on older variations of Lucene, simplifying future upgrades to the most recent model of OpenSearch Service. Whereas evaluating a transfer to OpenSearch 1 or 2, Octus chosen OpenSearch 1.3 because the goal to reduce client-side adjustments and cut back migration complexity, whereas positioning themselves for easier upgrades later.

The model choice notably impacted the R utility surroundings, as R language (an open-source programming language for statistical computing and knowledge evaluation) lacked native OpenSearch 1.3 consumer assist. This constraint required Octus to develop a customized consumer answer utilizing the ropensci/elastic library to combine with the brand new OpenSearch Service area. The Python surroundings introduced related challenges, the place the Elasticsearch 7.17 consumer constraints necessitated cautious consideration of the migration strategy. These consumer compatibility considerations have been among the many components that influenced the selection of Migration Assistant instruments over conventional snapshot-based strategies, because the Migration Assistant supplied higher assist for managing version-specific consumer interactions in the course of the transition.

Trying ahead, Octus plans to improve to newer OpenSearch variations as their utility stack evolves and consumer library assist matures, in order that they will leverage the most recent options and efficiency enhancements whereas sustaining the steadiness achieved by this migration.

Software modernization throughout a number of languages

The applying adjustments represented a major technical endeavor throughout a number of programming environments:

• Legacy PHP methods (5.6 and Laravel 4.2): Octus dealt with mapping sort deprecation on OpenSearch requests as specifying these mapping sorts usually are not supported, whereas persevering with to make use of the elasticsearch connector library with username/password authentication.
• Trendy PHP functions (8.1 and Laravel 9): These underwent extra complete adjustments, changing the elasticsearch/elasticsearch library with the opensearch-project/opensearch-php consumer and leveraging IAM authentication to connect with the clusters.
• Python surroundings: Purposes spanning variations 3.8, 3.10, 3.11, and three.13 with Django frameworks 2.1, 3.2, and 5.2 required changing the elasticsearch library with opensearch-py and transitioning to IAM authentication.
• R functions: For R 4.5.1 functions, Octus utilized a customized library ropensci/elastic to facilitate compatibility.

Site visitors routing and enhanced monitoring

To facilitate the migration, Octus redirected their present purchasers to route requests to the supply cluster by Migration Assistant’s Site visitors Seize Proxy, migrating the info from reside visitors to their goal cluster.

The monitoring infrastructure underwent vital enhancement throughout this course of. Octus’s observability infrastructure screens the general well being of OpenSearch Service clusters which incorporates cluster supervisor and knowledge nodes, community, knowledge storage, safety and IAM entry. It additionally screens the indexing and search efficiency of their functions. This alleviated the necessity for a separate monitoring cluster as logs and metrics have been shipped on to Datadog, considerably bettering observability. The Datadog screens have been outlined utilizing Infrastructure-as-Code and built-in seamlessly into their infrastructure frameworks.

Cutover and preliminary outcomes

The Web site Reliability Engineering staff meticulously deliberate the discharge, reaching a profitable migration from Elasticsearch to OpenSearch Service and cutover of the Elasticsearch consumer to the OpenSearch Service purchasers with no downtime for the system utility and nil knowledge loss. The preliminary migration part resulted in a 52% price discount whereas reaching operational advantages together with zero downtime for the system app, no knowledge loss, full Infrastructure-as-Code implementation for infrastructure and monitoring, and enhanced observability.

Put up-migration optimization

Following the migration, Octus carried out complete optimization based mostly on operational knowledge from manufacturing and different environments within the new OpenSearch Service setup. This real-world utilization knowledge supplied invaluable insights into precise useful resource consumption, enabling knowledgeable choices concerning additional cluster resizing.

By way of utilization metric evaluation and strategic resizing, Octus aligned cluster dimension extra exactly with operational wants, facilitating continued efficiency whereas minimizing expenditure. This optimization part delivered an extra 33% price discount in comparison with the unique Elastic Cloud prices, bringing the full discount to 85% whereas sustaining constant and optimum efficiency.

Operational monitoring

Octus makes use of Datadog to observe each search and indexing latency offering real-time visibility into Amazon OpenSearch Service cluster efficiency. The next screenshot showcases how customized Datadog dashboards present a reside view of the OpenSearch Service clusters. This visualization affords each a high-level overview and detailed insights into the ingestion course of, serving to us perceive the storage and doc rely. The underside half of the dashboard presents a time-series view of particular person node well being and efficiency metrics like learn and write latency, throughput and IOPS.

Determine 2 – DataDog dashboards

Migration observability

Migration Assistant for Amazon OpenSearch Service gives a number of dashboards to watch and validate the progress of a migration. Through the use of these observability options clients can monitor each backfill and reside seize and replay progress, facilitating confidence earlier than switching manufacturing workloads to the goal cluster.The next graphs are an instance from Octus’s migration, the place roughly 4TB of information was migrated in about 9 hours (from 08:00 to 17:00).

Determine 3 – Backfill progress by disk utilization

Determine 4 – Backfill progress by searchable paperwork

As soon as the backfill is full, the captured visitors is replayed to synchronize ongoing exercise between the supply and goal clusters.

On the time the backfill completed (round 17:00), the goal cluster was roughly 467 minutes behind the supply. The replay course of quickly decreased this lag by processing captured visitors at a quicker price than it was initially ingested on the supply.

Determine 5 – Replay lag after backfill completion

When the lag time reached 0, the goal cluster was absolutely in sync and manufacturing visitors might safely be rerouted. Octus selected to watch replayed visitors on the goal for a number of days earlier than making the ultimate switchover.

Reaching excellence

Octus’s migration to Amazon OpenSearch Service has yielded outstanding outcomes:

• Scalability – Octus has virtually doubled the variety of paperwork obtainable for Q&A throughout three environments in days as an alternative of weeks. Their use of Amazon Elastic Container Service (Amazon ECS) with AWS Fargate with auto scaling guidelines and controls offers them elastic scalability for his or her providers throughout peak utilization hours.
• Price discount – By shifting away from Elastic Cloud to OpenSearch Service, Octus’s month-to-month infrastructure prices are actually 85% decrease.
• Enhanced search efficiency – Octus maintained constant response instances all through the migration with no unfavorable impression on latency, whereas reaching a 20% enchancment in question throughput and total search efficiency.
• Zero downtime – Octus skilled zero downtime throughout migration and 100% uptime total for the entire utility.
• Lowered operational overhead – Put up-migration, Octus’s DevOps and SRE groups see 30% much less upkeep burden and overheads. Supporting SOC2 compliance can be easy now that they’re utilizing one system.
• Accelerated timeline supply – The whole migration was accomplished forward of schedule, shifting from planning to full completion in underneath one quarter.

“Shifting from Elastic Cloud to Amazon OpenSearch Service was a key element of our broader technique to reduce third-party dependencies and strengthen the reliability of Octus’ system infrastructure. Migration Assistant for Amazon OpenSearch Service enabled us to execute a seamless transition with zero knowledge loss and nearly no downtime for our customers.” – Vishal Saxena, CTO, Octus

Conclusion

On this put up, we confirmed you the way Octus efficiently migrated their Elasticsearch workloads from Elastic Cloud to Amazon OpenSearch Service utilizing the Migration Assistant for OpenSearch Service, reaching zero downtime and vital operational enhancements.

The Migration Assistant for OpenSearch Service supported this advanced migration by its complete suite of instruments. The Metadata Migration functionality migrated dozens of indices with numerous mappings and settings, with customized JavaScript transformations dealing with backward incompatibilities. Reindex-from-Snapshot migrated the historic paperwork from point-in-time snapshots with out impacting the supply cluster, whereas additionally optimizing the sharding scheme for improved efficiency. Stay Site visitors Replay made certain the goal cluster remained synchronized with real-time updates all through the migration course of.

The migration delivered substantial outcomes throughout the size. Octus achieved an 85% discount in month-to-month infrastructure prices whereas practically doubling the variety of paperwork obtainable for search throughout three environments. Search efficiency improved by 20% in question throughput with constant response instances and no unfavorable impression on latency. The migration maintained zero downtime and 100% uptime for all the utility, with DevOps and SRE groups experiencing 30% much less upkeep burden and operational overhead. The whole migration was accomplished forward of schedule in underneath one quarter.

To study extra concerning the Migration Assistant for OpenSearch Service and the way it might help you obtain related outcomes, go to the AWS Options dwelling web page.

Go to Octus to find out how we ship rigorously verified intelligence at velocity and create a whole image for professionals throughout all the credit score lifecycle. Observe Octus on LinkedIn and X.

Concerning the Authors