Benchmarking ideas

What matters are three factors:

• How much time does it take to resolve the data on the server?
• How much time does it take to download the data?
• How much time does it take to decrypt and apply the data?

The focus should be on two scenarios:

• Mobile device
• Desktop

For the mobile device a 3 Mbit/s (average 3G connection) should be simulated and the processor throttled to match a medium Android device. For the desktop device a 20 Mbit/s (slow cable connection) should be simulated.

There would be several test scenarios e.g.

• Load the Automerge paper recording of Martin Kleppmann https://github.com/josephg/diamond-types/tree/master/benchmark_data (opens in a new tab)
• Load the blog post recording of Martin Kleppmann https://github.com/josephg/diamond-types/tree/master/benchmark_data (opens in a new tab)
• Load 1 page document created by 1 client
• Load 10 page document created by 1 client
• Load 50 page document created by 1 client
• Load 1 page document created by 10 clients (two versions with and without simultaneous writes)
• Load 10 page document created by 10 clients (two versions with and without simultaneous writes)
• Load 50 page document created by 10 clients (two versions with and without simultaneous writes)
• Load 1 page document created by 100 clients (two versions with and without simultaneous writes)
• Load 10 page document created by 100 clients (two versions with and without simultaneous writes)
• Load 50 page document created by 100 clients (two versions with and without simultaneous writes)

A test would run multiple times and the median would be used for the result.