Are we heading toward a split between apache/java/zookeeper stacks and go/etcd on the other ? I've seen an issue related to that question on pulsar, and this got me investigating the distributed KV part of the stack.
It seems by looking at some benchmark that etcd is much more performant than zookeeper, and that to some people, operating two stacks seems like an operation maintenance cost a bit too high. Is that a valid concern ?
Also, i've seen that kafka is working on removing the dependency to zookeeper, is pulsar going to take the same road ?
This sound about right. Apart from maybe original Apache HTTP server most of the Apache projects are in Java.
Looking at codebase of Pulsar it looks like typical Apache style sprawling Java project with more than thousand directories, many thousand files and more than hundred dependencies. As comparison NATS which is in Go has few hundred files, less than hundred directories and about a dozen or so dependencies.
NATS is an amazing project, I just wanted to take the opportunity to highlight it for those first hearing about it in this comment. It's so brilliantly simple, yet changed the way I design distributed systems. I handle almost anything in regards to the standard messaging guarantees that a Kafka-like system offers at the endpoints now. As a result, systems are much simpler, and diagnosability of bugs or edge cases are much more straightforward.
NATS is amazing but note that it makes different promises than Pulsar. NATS doesn't offer true durability (in exchange for amazing performance and great simplicity) whereas Pulsar and similar are meant to survive certain partition or failure situations and not lose data.
It's not one or the other, they're just different tools.
There is nats-streaming-server as well which offers true durability (via file or SQL store) and a streaming model very similar to Kafka and Pulsar. It can also run as a raft cluster or in fault tolerance mode. It still has very good performance and is very simple to deploy and operate (I use it for event sourcing for real time IoT data at my day job).
NATS Streaming has major scalability problems even if it's simple to deploy. It's only high-availability unless you the Raft clustering but that has been bolted on to the original project and isn't really well-designed.
The team is working on an entirely new system called Jetstream to eventually replace it.
This sounds interesting, what exactly do you mean by 'endpoint' in this scenario? I looked into a few alternatives before settling for pulsar, and disregarded nats because it didn't seem to support message persistence. I didn't look into it too deeply though, maybe i should have. How do you guarantee no message is lost with NATS?
In my thinking, I think of an endpoint as something at either end of the communication channel (NATS in this case) where it is effectively terminal. Usually this is where the application logic lies. Dereck Collison (creator of NATS) brings this up in many of his talks about NATS, but I think the source of his thinking might come from “End-to-End Arguments in System Design” by Saltzer, Reed, & Clark.
The core of it is this point:
"Functions placed at low levels of a system may be redundant or of little value when compared with the cost of providing them at that low level."
That is, in order get that message redundancy or exactly once delivery, or message persistence, you pay a high cost, and you may be better off delegating to the endpoints.
"..Message/event persistence - NATS Streaming offers configurable message persistence: in-memory, flat files or database. The storage subsystem uses a public interface that allows contributors to develop their own custom implementations."
and
"At-least-once-delivery - NATS Streaming offers message acknowledgements between publisher and server (for publish operations) and between subscriber and server (to confirm message delivery). Messages are persisted by the server in memory or secondary storage (or other external storage) and will be redelivered to eligible subscribing clients as needed."
No, i missed that. I think ive seen 'nats streaming', but didn't realize that it is its own distinct thing. All this makes more sense now to me, thanks!
I think that the modern approach to distributed systems is moving towards golang style microservices and lightweight / simple system design with RPC communication, reconcile type loops for state reconciliation, and backing CP databases. I think this is the influence of k8s (and maybe google's approach to distributed systems).
I will almost certainly get downvoted for this (as I always seem to when I criticize the JVM), but Apache/JVM style architecture feels REALLY long in the tooth to me. I think you are committing to an outdated and very expensive approach to building software if you use anything running on the JVM, especially Apache based anything. Cassandra is a great example of this - out of the box it's a terribly performing database that is extremely expensive to run and tune. Throw enough resources and time at it and you can get it to acceptable scalability - but running on the JVM which is a huge memory hog will always make it expensive to run (and even then, you will always get terrible latency distributions with the JVM's awful GC).
If I was building a business I would run far far away from any JVM based solution. The only thing it has going for it is momentum. If you need to hire 100s of engineers off the street for a large project, then a JVM based stack is about your only option unfortunately.
This just makes it seem like you are trolling. JVM devs have done more to advance state of art in this area than any other language. The problem is that most JVM apps just produce too much garbage, not necessarily that the algo itself is awful.
Either way, there's no such thing as an optimal GC algorithm, just different trade-offs depending on your use case. Not everyone cares about latency.
I stand by that - I really think the JVM has a bad GC, and it has cost businesses billions probably. There is no trade off here - the JVM & Java have incredibly high memory requirements due to poor design - and this leads to all kinds of issues (like the bad GC latencies). No other language has this kind of problem as badly as the JVM does, even other GC languages. And we shouldn't accept this anymore - look at how much better golang's GC is for real world usage.
I understand your architecture criticism, and think it has merit, but I'm not sure why Apache gets dragged into that. Apache Airflow is in Python. Apache Arrow is in C. CouchDB is Erlang.
There's a ton of projects Apache Foundation hosts that fit your description but it's a mistake, I think, to confuse individual projects with Apache in general. Bad enough that people confuse the license with the foundation.
Sorry, you are right. It would probably be more accurate to use Spring actually. I was thinking of Cassandra/Zookeeper specifically when I said Apache.
Without knowing what that service does, how many users/request it serves per second/day, I can only assume it just an http listener up on some port that returns "Hello {username}" when someone sends a GET request.
Though criticizing the JVM is fair game, this sounds too absolutist. I like and dislike some things about C++, Java, Scala, Python, Bash, SQL, etc. I bet you do too, no?
I can't wait for projects to ditch ZooKeeper. Apache Bookkeeper, which Apache Pulsar uses for its state, already supports Etcd as a consensus store (though I believe this is still alpha? beta? quality). Pulsar is also working on supporting Etcd.
And I believe Kafka has a KIP in progress to remove their reliance on Zookeeper altogether but having the brokers communicate with each other directly.
It seems to be the trend. There would be more and more equivalents in Go in a foreseeable future. There would be a split.
I generally don't use Java or Go in my applications, but Go components are usually more lightweight and easier to use if you're not majorly working with those runtimes. JVM has a big overhead, and Java applications usually make things worse if they use things like dependency injection.
It depends. We have a ton of Java apps running atop of kubernetes. All of them use zk, but every team operates their own mini zk cluster deployed on k8s. It’s worked fine except for certain hard to debug problems that hit a few teams occasionally.
I guess my point is that k8s let’s you shift the operational burden to dev teams if they need it. If you have a centralized operations team running a giant, common zk/ etcd, yeah this would be additional operational burden.
Note that this is a different goal from the KIP referenced above, which is to entirely remove any dependency on an external configuration service. The idea of “pluggable consensus” is explicitly rejected in this KIP.
Are we heading toward a split between apache/java/zookeeper stacks and go/etcd on the other ? I've seen an issue related to that question on pulsar, and this got me investigating the distributed KV part of the stack.
It seems by looking at some benchmark that etcd is much more performant than zookeeper, and that to some people, operating two stacks seems like an operation maintenance cost a bit too high. Is that a valid concern ?
Also, i've seen that kafka is working on removing the dependency to zookeeper, is pulsar going to take the same road ?