AI
Why State Machines Break Under Distributed Scale
Online transaction processing updates financial data interactively and must complete in under 1 second. State machines model those updates cleanly in isolation. A single-node failure mid-transition leaves the current state unknown, stopping the business process. Copying the state machine across multiple nodes creates a new problem: any lost synchronization message causes nodes to diverge. Adding more replicas multiplies those failure points. An ATM stuck in that inconsistent state can retain a customer card, turning a software bug into a legal and financial problem.
The Consensus Tax That Blocks Microservice Scaling
Using a Raft-based store like etcd to share state across microservices makes the services themselves stateless and redundant. The problem is cost. Atomic broadcast is slow. Each additional microservice adds another round of consensus before the transaction can finish. That overhead accumulates inside a fixed response-time budget. As the service count grows, the consensus cost eventually becomes the scaling ceiling. Tuning storage parameters does not fix this. The transaction management layer itself is the bottleneck, and it needs to be rearchitected, not tuned.
PCT: Separating Coordination into a Dedicated Quorum
The Paxos Commit Transaction Orchestrator (PCT) pulls transaction management out of application middleware and into a standalone component. A PCT library annotating a business method sends a begin message to a coordinator quorum. When the method finishes, the library collects a readiness vote from every participating service and broadcasts those votes to the distributed coordinator group. The coordinators run Paxos commit to reach consensus. If one coordinator fails, the surviving majority still produces a consistent result. The commit decision returns to the database and the business method completes.
Independent Autoscaling for Business Services and Coordinators
PCT lets two distinct workload types scale on separate schedules. Business application pods, managed by Deployment, scale via HPA as transaction volume grows. The coordinator quorum, managed by StatefulSet, also scales via HPA independently. Because the final commit decision passes only through the coordinator group and not through every application service, the consensus path stays short. That separation is why consensus completes in about 10 milliseconds regardless of how many microservices participate in the transaction.
Reactive Processing Keeps Latency Flat as Services Grow
PCT uses Spring WebFlux to separate request and response threads, letting database updates run in parallel. Benchmark data shows the difference clearly. A non-reactive model inside the same PCT architecture produces response times that grow linearly as services are added. The reactive model holds response time flat at around 35 milliseconds as participant services scale from 2 to 8. The consensus contribution inside that 35-millisecond window stays constant across both models, confirming the Paxos commit path does not depend on microservice topology.
Notable Quotes
online transaction must be fast usually under 1 second but atomic broadcast is naturally slow and expensive as you add more microservices the cost of the saving state adopt this roll down uses up your limited time budget. Jumpei Nishitani · ▶ 12:29
You don’t solve the scaling issue by tuning the storage or consensus. You solve them by rearchitecting the transaction management itself. Jumpei Nishitani · ▶ 16:36
consensus reached in about 10 milliseconds every time Jumpei Nishitani · ▶ 22:00
even as the number of the application grow from the two to eight the response time is mostly flat at around 35 millisecond milliseconds. Jumpei Nishitani · ▶ 22:54
Key Takeaways
- Consensus cost per microservice, not storage tuning, is the real scaling ceiling in OLTP.
- PCT splits coordinator quorum (StatefulSet) and business services (Deployment) into independently HPA-scaled groups.
- Reactive processing with Spring WebFlux holds end-to-end latency flat at 35 ms across 2 to 8 services.
About the Speaker(s)
Jumpei Nishitani is a Senior Software Architect at Hitachi specializing in distributed systems and transaction processing. He began his career building HTTP servers and Jakarta EE technologies, and now leads architecture and technical evangelism for Hitachi’s microservices platform work.