Improve memory(storage) performance

[ellyxir]

Improve Memory(storage) Performance

Next steps:

• Implementation in 2 phases, transaction then cache (Irakli and Berni sync on this)
• CLI or not?
• Tab sync with IDB - any concerns?

[Gozala]

After in person discussion during onsite we have decided to do following:

1. Implement a new session interface that automatically subscribes to transaction log from the transaction provided as a parameter or from "last" transaction if parameter is no provided.
2. Use IDB to cache data locally so that it can be used in subsequent loads.
3. Use IDB to persist local (outgoing) updates, which are currently stacked in memory.
4. Use IDB to keep track of last seen transaction in order to resume subscription in next session.
5. Continue using existing query API in order to get latest state of relevant data when loading new charms.

[Gozala]

😥 I'm afraid this is going to take more time than I have originally projected. There are a lot more concurrency to deal with than I have anticipated which make implementation a lot more complex and nuanced. Specifically we have 3 tiers to worry about

1. State in active runtime (that only one writer updates)
2. State in the IDB (that multiple tabs can update concurrently)
3. State in the remote service (that multiple clients can update)

Complexity that I did not anticipate comes from the fact that tiers 1 and 2 can get desynchronized, because some tab can concurrency perform an update. My current design overlooked this and I'm afraid I have to revisit decisions around it to handle potential races better.

[Gozala]

I did extracted some work that I think we could land #938 which will be necessary down the line anyway.

[Gozala]

Implemented another supporting change on top of #938 that sends since with all queries #969 in order to be able to track fact timestamps in IDB cache, which is necessary to compare local version with received commit version.

[Gozala]

PR implementing all of this #989 and illustration of how it works

Currently everything seems to work, it is behind ENV variable flag and requires local toosheld instance (as it requires some changes in memory API).

I am still trying to make sure that all the edge cases work properly, specifically simulating network partition is proving difficult.

[Gozala]

#989 has merged landed in