Pitfalls in MongoDB Read-Write Separation


Problem Discovery and Troubleshooting

In the past few days, I developed a backend interface for updating user personal information POST /profile. The initial design was that after the frontend submitted the request, the backend would update the information and read the latest data from the database to return to the client. It's a simple write + read operation, which was quickly developed, passed unit testing, and launched in the test environment.

However, the testing team quickly told me that the data returned after each update was not the latest, but the old data from the previous time. This shouldn't be the case, as it had already passed the unit tests.

So I started to investigate. Initially, I suspected that the Promise in the code was being called asynchronously, and the problem was not discovered due to the fast execution speed of local unit tests. But this suspicion was quickly ruled out. Obviously, if it was called asynchronously, there would be no correct situation under the event loop mode.

Then I started to focus on the ODM used in the project. The MongoDB ODM I use is Mongoose. Because the native Mongoose should execute related logic through callback for different events. But when switching to the current async/await mode, I worry that the result of await is not the real call result. Or in other words, I suspect that Mongoose will send data operations to the MongoDB server and consider the request successful, so the caller may not be able to get the new data in real time. After a round of StackOverflow and a review of the documentation, I found that the update operation of Mongoose is indeed synchronous, and it will return the result only after MongoDB has indeed written the data.

Finally, I turned my attention to the database connection configuration, and eventually noticed a parameter readPreference=secondaryPreferred in the url of MongoDB. Then I realized that the delay was due to MongoDB master-slave replication. This could also explain why local unit tests could pass, but problems would occur in the online environment: because the local connection is not a database cluster, all reads and writes are in the master database, so there would be no delay.


Since we have identified the problem, we must consider how to solve it.

  • ❌ Placing read operations on the primary database

    Of course, this can solve all related problems in one step, but it loses the meaning of database read-write separation. When the number of read operations in the system far exceeds the number of write operations, this can lead to a decrease in system performance.

    Of course, you can also set it to select from the primary database when you need to get new data in real time. However, Mongoose does not support dynamic instance selection.

  • ⚠️ The update interface does not return in real time.

    In the Restful specification, it is recommended to return the updated entity after the update operation. However, on the one hand, not all update operations require immediate access to the latest data. On the other hand, if data consistency is to be ensured, the data obtained from the Get request should be used as the standard. The front end should not determine whether the update is successful based on whether the returned entity is compliant, but should be based on the returned 200 status code.

    Of course, designing the update interface to not return entities cannot completely solve the delay problem of master-slave replication. Because within the system, there are also many operations that generate new data in real time and then pull from the database. This situation requires a redesign.

    So, this solution still falls into the problem of treating the symptoms rather than the cause, and it is not universal.

  • ✅ Write Concern

    MongoDB also has a Write-Concern mechanism, which can flexibly adjust the distributed data read-write mechanism according to specific use cases.

    For write operations, Write-Concern provides multiple optional parameters.

    For read operations, the find operation has some interesting parameters.

    After all that, coming back to my use case, the more appropriate approach should be {w: "majority"} + {awaitData: true}