Berkeley DB Reference Guide: Network partitions

Berkeley DB Reference Guide:
Berkeley DB Replication

Network partitions

The Berkeley DB replication implementation can be affected by network partitioning problems.

For example, consider a replication group with N members. The network partitions with the master on one side and more than N/2 of the sites on the other side. The sites on the side with the master will continue forward, and the master will continue to accept write queries for the databases. Unfortunately, the sites on the other side of the partition, realizing they no longer have a master, will hold an election. The election will succeed as there are more than N/2 of the total sites participating, and there will then be two masters for the replication group. Since both masters are potentially accepting write queries, the databases could diverge in incompatible ways.

If multiple masters are ever found to exist in a replication group, a master detecting the problem will return DB_REP_DUPMASTER. If the application sees this return, it should reconfigure itself as a client (by calling DB_ENV->rep_start), and then call for an election (by calling DB_ENV->rep_elect). The site that wins the election may be one of the two previous masters, or it may be another site entirely. Regardless, the winning system will bring all of the other systems into conformance.

As another example, consider a replication group with a master environment and two clients A and B, where client A may upgrade to master status and client B cannot. Then, assume client A is partitioned from the other two database environments, and it becomes out-of-date with respect to the master. Then, assume the master crashes and does not come back on-line. Subsequently, the network partition is restored, and clients A and B hold an election. As client B cannot win the election, client A will win by default, and in order to get back into sync with client B, possibly committed transactions on client B will be unrolled until the two sites can once again move forward together.

In both of these examples, there is a phase where a newly elected master brings the members of a replication group into conformance with itself so that it can start sending new information to them. This can result in the loss of information as previously committed transactions are unrolled.

In architectures where network partitions are an issue, applications may want to implement a heart-beat protocol to minimize the consequences of a bad network partition. As long as a master is able to contact at least half of the sites in the replication group, it is impossible for there to be two masters. If the master can no longer contact a sufficient number of systems, it should reconfigure itself as a client, and hold an election. Replication Manager does not currently implement such a feature, so this technique is only available to applications which use the Base replication API.

There is another tool applications can use to minimize the damage in the case of a network partition. By specifying an nsites argument to DB_ENV->rep_elect that is larger than the actual number of database environments in the replication group, applications can keep systems from declaring themselves the master unless they can talk to a large percentage of the sites in the system. For example, if there are 20 database environments in the replication group, and an argument of 30 is specified to the DB_ENV->rep_elect method, then a system will have to be able to talk to at least 16 of the sites to declare itself the master.

Replication Manager uses the value of nsites (configured by the DB_ENV->rep_set_nsites method) for elections as well as in calculating how many acknowledgements to wait for when sending a DB_REP_PERMANENT message. So this technique may be useful here as well, unless the application uses the DB_REPMGR_ACKS_ALL or DB_REPMGR_ACKS_ALL_PEERS acknowledgement policies.

Specifying a nsites argument to DB_ENV->rep_elect that is smaller than the actual number of database environments in the replication group has its uses as well. For example, consider a replication group with 2 environments. If they are partitioned from each other, neither of the sites could ever get enough votes to become the master. A reasonable alternative would be to specify a nsites argument of 2 to one of the systems and a nsites argument of 1 to the other. That way, one of the systems could win elections even when partitioned, while the other one could not. This would allow one of the systems to continue accepting write queries after the partition.

In a 2-site group, Replication Manager reacts to the loss of communication with the master by assuming the master has crashed: the surviving client simply declares itself to be master. Thus it avoids the problem of the survivor never being able to get enough votes to prevail. But it does leave the group vulnerable to the risk of multiple masters, if both sites are running but cannot communicate.

These scenarios stress the importance of good network infrastructure in Berkeley DB replicated environments. When replicating database environments over sufficiently lossy networking, the best solution may well be to pick a single master, and only hold elections when human intervention has determined the selected master is unable to recover at all.