Yesterday, I wrote about the use of the class TransactionExecutor. Today, I'm going to propose a new way of using it within Socorro's HBase crashstorage subsystem.
When used with Postgres, the TransactionExecutor treats all of the individual steps within a transaction collectively as if they were together a single atomic operation. This is the exactly how we want a database transaction treated. Lets say we have a transaction that consists of steps A, B, C, D. We start a transaction and succeed in steps A and B. However, C fails. The TransactionExecutor rolls back the transaction, essentially undoing steps A and B. Then it retries the whole thing from the beginning after sleeping.
That idea isn't ideal for HBase. Since HBase doesn't support relational database-like transactions, the same strategy of treating multiple steps as one atomic transaction doesn't work as well. Just like the previous example, lets say that for HBase we have four steps A, B, C and D. We start a “transaction” and complete steps A and B before we get a failure in step C. If the transaction executor is used exactly like it is in Postgres, we rollback the transaction. But HBase doesn't support transactions, so steps A and B are not actually undone. The TransactionExecutor then sleeps and after waking, retries the whole thing from the beginning.We end up redoing steps A and B that have already been done.
What are the consequences of this? It is my understanding that rows in tables in HBase are static. We don't really change them, we just make a new version of the row. The old version of the row still sticks around (the number of old rows that HBase saves is configurable). By starting the transaction over from the beginning, we're increasing the amount of space that HBase needs to save the same information.
Honestly, this isn't much of a problem. During a failure, having multiple copies of the same information will happen only to a single row (or in our case, one row per actively writing thread during the HBase problem). That's not a lot of extra storage wasted in comparison to the huge size of our data storage.
However, if we care to, we can resolve this duplication problem by using the TransactionExecutor in a different manner. Rather than treating all the steps A, B, C, D as if they are collectively atomic, we could use the TransactionExecutor individually on each step. If we don't fail until step C, then only step C will be retried. We can avoid the duplication of steps A and B.
For purity sake, it makes sense to use the TransactionExecutor on operations that are truly atomic. In our application with HBase, it really doesn't matter too much and would be likely more trouble than it is worth to change.