10 Managing Data Store Failover and Recovery

This chapter applies to all replication schemes, including active standby pairs. However, TimesTen integration with Oracle Clusterware is the best way to monitor active standby pairs. See Chapter 6, "Using Oracle Clusterware to Manage Active Standby Pairs".

This chapter includes these topics:

Overview of data store failover and recovery
General failover and recovery procedures
Recovering a failed data store
Recovering nondurable data stores
Writing a failure recovery script

Overview of data store failover and recovery

A fundamental element in the design of a highly available system is the ability to quickly recover from a failure. Failures may be related to hardware problems such as system failures or network failures. Software failures include operating system failure, application failure, data store failure and operator error.

Your replicated system must employ a "cluster manager" or custom software to detect such failures and, in the event of a failure involving a master data store, redirect the user load to one of its subscribers. TimesTen does not provide a cluster manager or make any assumptions about how they operate, so the focus of this discussion is on the TimesTen mechanisms that an application or cluster manager can use to recover from failures.

Unless the replication scheme is configured to use the return twosafe service, TimesTen replicates updates only after the original transaction commits to the master data store. If a subscriber data store is inoperable or communication to a subscriber data store fails, updates at the master are not impeded. During outages at subscriber systems, updates intended for the subscriber are saved in the TimesTen transaction log.

Note:

The procedures described in this section require the ADMIN privilege.

General failover and recovery procedures

The procedures for managing failover and recovery depend primarily on:

The replication scheme
Whether the failure occurred on a master or subscriber data store
Whether the threshold for the transaction log on the master is exhausted before the problem is resolved and the data stores reconnected

Subscriber failures

In a default asynchronous replication scheme, if a subscriber data store become inoperable or communication to a subscriber data store fails, updates at the master are not impeded and the cluster manager does not have to take any immediate action.

Note:

If the failed subscriber is configured to use a return service, you must first disable return service blocking, as described in "Managing return service timeout errors and replication state changes"6.

During outages at subscriber systems, updates intended for the subscriber are saved in the transaction log on the master. If the subscriber agent reestablishes communication with its master before the master reaches its FAILTHRESHOLD, the updates held in the log are automatically transferred to the subscriber and no further action is required. See "Setting the log failure threshold" for details on how to establish the FAILTHRESHOLD value for the master data store.

If the FAILTHRESHOLD is exceeded, the master sets the subscriber to the Failed state and it must be recovered, as described in "Recovering a failed data store". Any application that connects to the failed subscriber receives a tt_ErrReplicationInvalid (8025) warning indicating that the data store has been marked Failed by a replication peer.

An application can use the ODBC SQLGetInfo function to check if the subscriber data store it is connected to has been set to the Failed state. The SQLGetInfo function includes a TimesTen-specific infotype, TT_REPLICATION_INVALID, that returns a 32-bit integer value of '1' if the data store is failed, or '0' if not failed. Since the infotype TT_REPLICATION_INVALID is specific to TimesTen, all applications using it need to include the timesten.h file in addition to the other ODBC include files.

Example 10-1 Checking whether a data store has been set to the Failed state

Check if the data store identified by the hdbc handle has been set to the Failed state.

SQLINTEGER retStatus;

SQLGetInfo(hdbc, TT_REPLICATION_INVALID,
          (PTR)&retStatus, NULL, NULL);

Master failures

The cluster manager plays a more central role if a failure involves the master data store. If a master data store fails, the cluster manager must detect this event and redirect the user load to one of its surviving data stores. This surviving subscriber then becomes the master, which continues to accept transactions and replicates them to the other surviving subscriber data stores. If the failed master and surviving subscriber are configured in a bidirectional manner, transferring the user load from a failed master to a subscriber does not require that you make any changes to your replication scheme. However, when using unidirectional replication or complex schemes, such as those involving propagators, you may have to issue one or more ALTER REPLICATION statements to reconfigure the surviving subscriber as the "new master" in your scheme. See "Replacing a master data store" for an example.

When the problem is resolved, if you are not using the bidirectional configuration or the active standby pair described in "Automatic catch-up of a failed master data store", you must recover the master data store as described in "Recovering a failed data store".

After the data store is back online, the cluster manager can either transfer the user load back to the original master or reestablish it as a subscriber for the "acting master."

Automatic catch-up of a failed master data store

The master catch-up feature automatically restores a failed master data store from a subscriber data store without the need to invoke the ttRepAdmin -duplicate operation described in "Recovering a failed data store".

The master catch-up feature needs no configuration, but it can be used only in the following types of configurations:

A single master replicated in a bidirectional manner to a single subscriber
An active standby pair in which the active data store is replicated to the standby data store which then propagates changes to up to 127 read-only subscribers

In addition, the following must be true:

The ELEMENT type is DATASTORE.
TRANSMIT NONDURABLE or RETURN TWOSAFE must be enabled. TRANSMIT NONDURABLE is optional for asynchronous and return receipt transactions.
All replicated transactions must be committed nondurably. They must be transmitted to the other master data store before they are committed on the local one. For example, if the replication scheme is configured with RETURN TWOSAFE BY REQUEST and any transaction is committed without first enabling RETURN TWOSAFE, master catch-up may not occur after a failure of the master.

When the master replication agent is restarted after a crash or invalidation, any lost transactions that originated on the master are automatically reapplied from the subscriber to the master. No connections are allowed to the master store until it has completely caught up with the subscriber. Applications attempting to connect to a data store during the catch-up phase receive an error that indicates a catch-up is in progress. The only exception is if you connect to a data store with the ForceConnect attribute set in the DSN.

When the catch-up phase is complete, your application can connect to the data store. An SNMP trap and message to the system log indicate the completion of the catch-up phase.

If one of the stores is invalidated or crashes during the catch-up process, the catch-up phase is resumed when the store comes back up.

Failures in bidirectional distributed workload schemes

You can distribute the workload over multiple bidirectionally replicated data stores, each of which serves as both master and subscriber. When recovering a master/subscriber data store, the log on the failed data store may present problems when you restart replication. See "Bidirectional distributed workload scheme".

If a data store in a distributed workload scheme fails and work is shifted to a surviving data store, the information in the surviving data store becomes more current than that in the failed data store. If replication is restarted at the failed system before the FAILTHRESHOLD has been reached on the surviving data store, then both data stores attempt to update one another with the contents of their transaction logs. In this case, the older updates in the transaction log on the failed data store may overwrite more recent data on the surviving system.

There are two ways to recover in such a situation:

If the timestamp conflict resolution rules described in Chapter 13, "Resolving Replication Conflicts" are sufficient to guarantee consistency for your application, then you can restart the failed system and allow the updates from the failed data store to propagate to the surviving data store. The conflict resolution rules prevent more recent updates from being overwritten.
Re-create the failed data store, as described in "Recovering a failed data store".

Note:
If the data store must be re-created, the updates in the log on the failed data store that were not received by the surviving data store cannot be identified or restored. In the case of several surviving data stores, you must select which of the surviving data stores is to be used to re-create the failed data store. It is possible that at the time the failed data store is re-created, that the selected surviving data store may not have received all updates from the other surviving data stores. This results in diverging data stores. The only way to prevent this situation is to re-create the other surviving data stores from the selected surviving data store.

Network failures

In the event of a temporary network failure, you need not perform any specific action to continue replication. The replication agents that were in communication attempt to reconnect every few seconds. If the agents reconnect before the master data store runs out of log space, the replication protocol makes sure they neither miss nor repeat any replication updates. If the network is unavailable for a longer period and the FAILTHRESHOLD has been exceeded for the master log, you need to recover the subscriber as described in "Recovering a failed data store".

Failures involving sequences

After a link failure, if replication is allowed to recover by replaying queued logs, you do not need to take any action.

However, if the failed node was down for a significant amount of time, you must use the ttRepAdmin -duplicate command to repopulate the data store on the failed node with transactions from the surviving node, as sequences are not rolled back during failure recovery. In this case, the ttRepAdmin -duplicate command copies the sequence definitions from one node to the other.

Recovering a failed data store

If the data stores are configured in a bidirectional replication scheme, a failed master data store is automatically brought up to date from the subscriber. See "Automatic catch-up of a failed master data store". Automatic catch-up also applies to recovery of master data stores in active standby pairs.

If a restarted data store cannot be recovered from its master's transaction log so that it is consistent with the other data stores in the replicated system, you must re-create the data store from one of its replication peers. Use command line utilities or the TimesTen Utility C functions. See "Recovering a failed data store from the command line" and "Recovering a failed data store from a C program".

Note:

It is not necessary to re-create the DSN for the failed data store.

In the event of a subscriber failure, if any tables are configured with a return service, commits on those tables in the master data store are blocked until the return service timeout period expires. To avoid this, you can establish a return service failure and recovery policy in your replication scheme, as described in"Managing return service timeout errors and replication state changes". If you are using the RETURN RECEIPT service, an alternative is to use ALTER REPLICATION and set the NO RETURN attribute to disable return receipt until the subscriber is restored and caught up. Then you can submit another ALTER REPLICATION statement to re-establish RETURN RECEIPT.

Recovering a failed data store from the command line

If the data stores are fully replicated, you can use ttDestroy to remove the failed data store from memory and ttRepAdmin -duplicate to re-create it from a surviving data store. If the data store contains any cache groups, you must also use the -keepCG option of ttRepAdmin. See "Duplicating a data store".

Example 10-2 Recovering a failed data store

To recover a failed data store, subscriberds, from a master, named masterds on host system1, enter:

> ttdestroy /tmp/subscriberds

> ttrepadmin -dsn subscriberds -duplicate -from masterds -host "system1" -uid ttuser

You will be prompted for the password of ttuser.

Note:

ttRepAdmin -duplicate is only supported between identical and patch TimesTen releases. The major and minor release numbers must be the same.

After re-creating the data store with ttRepAdmin -duplicate, the first connection to the data store reloads it into memory. To improve performance when duplicating large data stores, you can avoid the reload step by using the ttRepAdmin -ramload option to keep the data store in memory after the duplicate operation.

Example 10-3 Keeping a data store in memory when recovering it

To recover a failed data store, subscriberds, from a master, named masterds on host system1, and to keep the data store in memory and restart replication after the duplicate operation, enter:

> ttdestroy /tmp/subscriberds

> ttrepadmin -dsn subscriberds -duplicate -ramload -from masterds -host "system1" -uid ttuser -setmasterrepstart

You will be prompted for the password of ttuser.

Note:

After duplicating a data store with the ttRepAdmin -duplicate -ramLoad options, the RAM Policy for the data store is manual until explicitly reset by ttAdmin -ramPolicy or the ttRamPolicy function.

Recovering a failed data store from a C program

You can use the C functions provided in the TimesTen utility library to recover a failed data store programmatically.

If the data stores are fully replicated, you can use ttDestroyDataStore function to remove the failed data store and the ttRepDuplicateEx function to re-create it from a surviving data store.

Example 10-4 Recovering and starting a failed data store

To recover and start a failed data store, named subscriberds on host system2, from a master, named masterds on host system1, enter:

int          rc;
ttutilhandle utilhandle;
ttrepduplicateexarg arg;
memset( &arg, 0, sizeof( arg ) );
arg.size = sizeof( ttrepduplicateexarg );
arg.flags = tt_repdup_repstart | tt_repdup_ramload;
arg.uid=ttuser;
arg.pwd=ttuser;
arg.localhost = "system2";
rc = ttdestroydatastore( utilhandle, "subscriberds", 30 );
rc = ttrepduplicateex( utilhandle, "dsn=subscriberds",
                      "masterds", "system1", &arg );

In this example, the timeout for the ttDestroyDataStore operation is 30 seconds. The last parameter of the ttRepDuplicateEx function is an argument structure containing two flags:

TT_REPDUP_RESTART to set the subscriberds data store to the start state after the duplicate operation is completed
TT_REPDUP_RAMLOAD to set the RAM policy to manual and keep the data store in memory

Note:

When the TT_REPDUP_RAMLOAD flag is used with ttRepDuplicateEx, the RAM policy for the duplicate data store is manual until explicitly reset by the ttRamPolicy function or ttAdmin -ramPolicy.

See "TimesTen Utility API" in Oracle TimesTen In-Memory Database C Developer's Guide for the complete list of the functions provided in the TimesTen C language utility library.

Recovering nondurable data stores

If your data store is configured with the TRANSMIT NONDURABLE option in a bidirectional configuration you do not need to take any action to recover a failed master data store. See "Automatic catch-up of a failed master data store".

For other types of configurations, if the master data store configured with the TRANSMIT NONDURABLE option fails, you must use ttRepAdmin-duplicate or ttRepDuplicateEx to re-create the master data store from the most current subscriber data store. If the application attempts to reconnect to the master store without first performing the duplicate operation, the replication agent recovers the data store, but any attempt to connect results in an error that advises you to perform the duplicate operation. To avoid this error, the application must reconnect with the ForceConnect connection attribute set to 1.

Writing a failure recovery script

Upon detecting a failure, the cluster manager should invoke a script that effectively executes the procedure shown by the pseudocode in Example 10-5.

Example 10-5 Failure recovery pseudocode

Detect problem {
       if (Master == unavailable) {
          FailedDataStore = Master
          FailedDSN = Master_DSN
          SurvivorDataStore = Subscriber
          switch users to SurvivorDataStore
      }
else {
          FailedDataStore = Subscriber
          FailedDSN = Subscriber_DSN
          SurvivorDataStore = Master
      }
}
Fix problem....
If (Problem resolved) {
       Get state for FailedDataStore
       if (state == "failed") {
         ttDestroy FailedDataStore
         ttRepAdmin -dsn FailedDSN -duplicate
                 -from SurvivorDataStore -host SurvivorHost
                 -setMasterRepStart
                 -uid ttuser
                 -pwd ttuser
      }
      else {
         ttAdmin -repStart FailedDSN
      }
      while (backlog != 0) {
         wait
      }
}

Switch users back to Master.

This applies to either the master or subscriber data stores. If the master fails, you may lose some transactions.