Extending transactional guarantees of common-memory

[Gozala]

Goal

Implement transactions that can:

1. Perform several entity updates
2. Ensure entity invariants (current state)
3. Succeed all or fail all

Deliverable

Once done storage provider will start sending update batches as opposed to individual document updates.

Details

Had a white-boarding session with @seefeldb today (see below) where we identified several new requirements for common-memory store

Transactions

1. We want to add atomic transaction, that is set of updates that must be all applied or all rejected.
2. We also want to chart a way for a Software Transactional Memory (STM) system that we want to implement next. Implication here is that transactions in addition to capturing what entity changed from what state we also want to capture what we assumed state of some entities were even though they aren't updated.
3. We want to capture and store transactions in the store also so that eventually we could traverse the whole transaction log.

[Gozala]

I am not sure we necessarily need to capture reads in the transactions, that is because they could be expressed as noop writes, specifically if you submit an assertion who's conclusion matches current state it is not rejected, because it has no consequence, I think we could simply capture all reads as such, this way if read data has changed since it will be rejected but if it has not update ill be of no consequence.

[Gozala]

In synopsys we represented transactions as commit entities and stored those in the DB I think we want to do something similar here. We also want to associate latest commit to some (I think db) entity so it is possible to traverse it if desired.

I think interesting question is should entity be the same across all DBs or should it be unique & potentially derived from the DB identifier. Later seems like safer choice (e.g. if for some reason in the future we needed to store transaction info across instances), but it would make querying history bit more nuanced.

[Gozala]

I think interesting question is should entity be the same across all DBs or should it be unique & potentially derived from the DB identifier. Later seems like safer choice (e.g. if for some reason in the future we needed to store transaction info across instances), but it would make querying history bit more nuanced.

Actually we did support forking / branching in the future, differentiating here would be helpful, so I think deriving id from replica seems like a better choice.

[Gozala]

Interesting question that I'll attempt clarify with @seefeldb is whether we want transactional guarantees across db instances. I personally don't like the idea of it, but perhaps there are product level requirements for this.

[Gozala]

Interesting question that I'll attempt clarify with @seefeldb is whether we want transactional guarantees across db instances. I personally don't like the idea of it, but perhaps there are product level requirements for this.

We discussed this in person yesterday, and decided that no guarantees across db instances are required.

[Gozala]

One thing that we identified also yesterday, was desire to add lamport timestamps or perhaps hybrid logical clock timestamps in order to perform queries in time. More specifically we may have updated document A at timestamp 2 and then at timestamp 9, what if we want to ask what was the state of document A at timestamp 5 ?

Ordering by cause implies walking the chain which would make ☝️ complicated. But if we capture timestamps we could have query like

SELECT *
FROM fact
WHERE timestamp = (
    SELECT MAX(timestamp)
    FROM fact
    WHERE timestamp < :timestamp
);

[Gozala]

Was fixed by #349