Synchronizing DocumentDB and OpenSearch Data Using Monstache

Olive Young manages denormalized data stored in AWS DocumentDB to consolidate distributed product data. However, to solve the load problem that occurs during large-scale data queries, they adopted an architecture that synchronizes data to OpenSearch.

The open source they chose for data synchronization is Monstache. This tool is based on the Go language and detects the oplog of MongoDB (DocumentDB) to transmit data to OpenSearch in real time. Its key features include real-time synchronization, filtering and transformation, scripting support, and status checking via a web server.

The key configuration and optimization aspects during implementation are as follows.

• Enhanced Security: When managing AWS credentials, they increased security by using an EC2 IAM Role (strategy = 4) instead of an Access Key.
• Timeout Adjustment: To resolve the health check timeout issue that occurred during OpenSearch connections, they secured stability by increasing the elasticsearch-healthcheck-timeout value to 200.
• Enabling Change Streams: They enabled the Change Streams feature in DocumentDB and applied the readPreference=primary option to build a real-time detection environment.

They are currently operating the synchronization system stably, performing error log detection and health check monitoring through Datadog.