One company I was at sold its product claiming any changes you made were “near instantly published” globally. They tried to demo it as such.
The way the engineers built the update/publish operation was synchronous from their primary data center to a number of globally distributed data centers. Publish didn’t “complete” until a receiver in each data center responded with an ACK after parsing and uploading to a nearby region cloud bucket. Any failure/timeout caused the entire transaction across all data centers to retry. All of traffic was over multiple VPNs, hub and spoke style. They built this system in 2020.
They constantly complained and generated incident reports about p95/p99 latencies to the Asia regions. Latencies that were perfectly reasonable when you considered the multiple global round trips that were being made, the size/volume of objects in the publish, set of operations and speed of light.
They swore that because the client UX to publish the change to the primary data center used JavaScript async that the entire process was async. They denied repeatedly that their “all receivers ack complete to succeed” business logic was synchronous. I shit you not.
The way the engineers built the update/publish operation was synchronous from their primary data center to a number of globally distributed data centers. Publish didn’t “complete” until a receiver in each data center responded with an ACK after parsing and uploading to a nearby region cloud bucket. Any failure/timeout caused the entire transaction across all data centers to retry. All of traffic was over multiple VPNs, hub and spoke style. They built this system in 2020.
They constantly complained and generated incident reports about p95/p99 latencies to the Asia regions. Latencies that were perfectly reasonable when you considered the multiple global round trips that were being made, the size/volume of objects in the publish, set of operations and speed of light.
They swore that because the client UX to publish the change to the primary data center used JavaScript async that the entire process was async. They denied repeatedly that their “all receivers ack complete to succeed” business logic was synchronous. I shit you not.