MongoDB read-write separation pitfalls

Troubleshooting¶

These two days I developed a backend interface to update user profile information POST/profile. The initial design was that after the front end submitted a request, the backend would update the information and read the latest data from the database, and return it to the client. It was a very simple write + read operation, and it was quickly developed and passed the unit test and went live to the test environment.

However, the tester soon told me that the data returned was not the latest every time after the update, but the old data from the last time. It shouldn't be like this, because it has passed the unit test.

So we started to troubleshoot. At first, we suspected that the Promise in the code was called asynchronously, and the local unit test executed too fast to find the problem. But this problem was quickly ruled out. Obviously, if it was called asynchronously, there would be no correct case in the eventloop mode.

Then I started to focus on the ODM used in the project. The MongoDBODM I used is Mongoose. Because the original Mongoose should execute the relevant logic by performing a callback on different events, but switching to the current async/await mode, I was worried that the result of await would not be the real result of the call. Or, I suspected that Mongoose would send the data operation to the MongoDB server and think that the request was successful, then the caller should not be able to get the new data in real time. I searched StackOverflow and went through the documentation, and found that Mongoose's update operation is indeed synchronous. It must be that MongoDB actually wrote the data before returning the result.

Finally, I turned my attention to the database connection configuration, and finally noticed that there was a parameter readPreference=secondaryPreferred in the MongoDB url. Now I suddenly realized that the delay in the middle appeared in MongoDB master-slave replication. In this way, it can also explain why local unit tests can pass, while online environments will have problems: because the local connection is not a database cluster, all reads and writes exist in the primary database, so there will be no delay.

Solution¶

Now that we have found the problem, we need to consider how to solve it

• ❌ Put read operations on the primary

  This can certainly solve all the related problems in one step, but it defeats the purpose of database read-write separation. When the read operations in the system far exceed the write operations, this will cause the system performance to decrease.

  Of course, you can also choose to read from the primary instance when you need to get new data in real time. However, Mongoose does not support dynamic instance selection.

• ⚠️update interface does not return in real time

  In Restful specification, it is recommended to return the updated entity after the update operation. However, not all update operations need to get the latest data immediately. On the other hand, if data consistency needs to be guaranteed, the data obtained by the Get request should be used as the standard. The front end should not determine whether the update is successful based on whether the returned entity is consistent, but should use the returned 200 status code as the standard.

  Of course, designing the update interface to not return entities cannot completely solve the problem of master-slave replication delay. Because within the system, there are also many operations that generate new data in real time and then pull it from the database. This situation requires redesign.

  So this solution still falls into the problem of treating the symptoms but not the root cause, and it is not universally applicable.

• WriteConcern

  MongoDB also has a Write-Concern mechanism, which can flexibly adjust the distributed data and read-write mechanism according to the specific usage scenario.

  For write operations, Write-Concern provides multiple optional parameters.

  • You can use {w:<X>} to specify how many instances need to synchronize data before returning a successful confirmation.

    The default is {w:1}, which means the data will be returned after the primary node confirms.

    Can be set to {w:"majority"}, to ensure that most instances are confirmed before returning.

    Of course, it can also be set to {w:0}, just like I said above, the operation is returned as soon as it is sent, without waiting for write confirmation, more like caching, and it doesn't matter even if data is lost, the key is that it is fast.

  • By using {j:boolean}, ensure that the data is returned after it is written to disk. It is turned off by default. Generally, MongoDB will put the data in memory and write it to disk when appropriate. However, if you have high requirements for data security, you can turn it on.
  • If {w:"majority"} is used, there may be a problem with synchronization failure between nodes. We can't wait forever. wtimeout can limit the time it takes to return a write failure.

  For read operations, the find operation has some interesting parameters

  • readConcern: It can limit the read strategy. The default is local, which means reading data from the current node.

    It can also be set to majority, which means that the data of most nodes has been successfully written, corresponding to the above {w:"majority"}. But in fact, the final read is still the current node, but if the current node has not synchronized the oplog with the primary, there is no need to talk about whether it is the latest data. Therefore, majority basically guarantees that the data is confirmed by the entire cluster and will not be returned data.

  • You can set {awaitData: }, this parameter allows the find operation to wait. If the corresponding data is not found after a certain period of time, it is considered non-existent. This period of time can be used as the delay for synchronization between nodes.

  Speaking of my use case, a more appropriate approach should be {w:"majority"}+{awaitData:true}