I worked on Google Fiber's TV distribution system so know something about this although GFiber's TV offering was IPv4 multicast not IPv6.
For anyone unfamiliar, modern video codecs tend to break up a video into a group of pictures (GOP) and you have a mix of different frame types. Some of these are the entire image. Most are just differences. It gets more complex as the video codecs allow you to pull objects from previous and future frames. But I digress.
The point is one of the things you have to decide is how often you send a complete image. Typically that's every 1-2 seconds. It may be more often with scene changes. The bigger the minimum gap in full images, the lower the bandwidth. The downside? The longer it takes to start playing a channel. Also, it makes network interruptions worse as you may have distorted the image and sound for up to 2 seconds instead of 1.
You also have to make a choice between CBR (constant bit rate) and VBR (variable bit rate). CBR is often preferred for mass distribution because it doesn't overload what are usually lower powered CPUs but that will also lead to a delay in channel changes as a full image may take longer than 1/24 or 1/30 of a second to actually send.
And then when you get into sending video over a network you have to worry about some Russian dolls of various standards. The video has a format. So does the audio. Those are in a container format (eg MKV). And then you have a transport format. For Fiber this was (and probably still is but I'm guessing) MPEG-TS (Transport Streams), which is a weird format that goes all the way back to ATM. Basically 7 packets were put into a single multicast IPv4 packet. Why 7? Because it's the most you could fit while staying under an MTU of 1500. As anyone who has dealt with IP, things just stop working or get more difficult with large MTUs.
Now on top of all of that and minimum 2-3 second channel switches you need to deal with multicast channel subscription delays. I don't know if this was changed with IPv6 but IPv4 multicast just wasn't designed with low latency subscription changes in mind. This goes to an underlying philosophy TCP/IP had in its inception of at-most once delivery. In the modern world, we've come to realize that's wrong for streaming content and we want at-least once delivery instead.
But it gets worse than that. When playing video you generally want to have a reasonable buffer. This way you have time to recover if you have some lost or damaged packets. Multicast doesn't give you a way of dealing with that so you need to build a whole different set of infrastructure for that.
Video playback in general is really difficult to get right, reliably. As an eample, my Chrome for reasons I haven't been able to ascertain, stutters every second when playing certain csources. Firefox doesn't. I suspect it's something to do with the aofrementioned full image frame every second. If Google can't get that right, is it any wonder that so many problems exist?
Circling back, multicast is a really awkward fit for something like this and it's not going to be a great user experience but, somewhat surprisingly, the alternatives still have issues.
Take for example DASH. This is a more modern method of video transport except it's kind of weird because you have an XML manifest file. This works quite well for VODs. There are space-saving variants that allow for essentially templated frame names rather than listing each frame and stream idnividually. But when you get to streaming live video, it gets rather awkward because the client may end up requesting a segment that doesn't exist yet. What should you do? Do you block the HTTP request and wait for it? Do you return a 404?
I guess this is a really long-winded way of saying I'd never do multicast distribution unless I had to. Or I was just playing around. It would be interesting to see what devices just absolutely choke when trying to consume multicast IPv6 video and the various container and stream formats that may entail.
I have to disagree with this as someone who worked at Meta (and Google).
WhatsApp was a textbook example of how not to do an acquisition. The story I heard was when it was acquired, a spreadsheet went around and everyone basically decided what level they were in the (then) FB job ladder and all the engineers said they were E7s (Senior Staff SWE). The way PSC worked, WhatsApp at the time was only ever calibrated against themselves (from what I heard). It had become a fiefdom and, as someone who was on a team that tried to get them to do anything, the experience was awful.
IG was handled better but it was also an almost nonexistent team when acquired, which might well explain it. They stuck with their Django/Python codebase and (IMHO) that was a mistake. The amount of duplication that we had to do for IG specifically was embarrasing. The framework and tooling FB had on the product side was light years ahead of what IG had. IG used to have a very good product focus but I think that's long dead now. It was good because IG had a clear vision for their app and ultimately (IMHO) management had a different view to "grow". They briefly tried to launch another app (IGTV) that flopped, hard. There were a bunch of UI/UX changes that clearly showed the focus had become simply following celebrities instead of sharing updates (eg where the post/compose buttons moved to).
I mention Google because I saw the same things happen at Google.
Youtube was (and my guess is, still is) its own entity. Culturally, Youtubers don't see themselves as Googlers. They didn't (AFAIK) use Google3 or any of the other stuff most of the rest of Google did. But Youtube itself was perceived very positively, technically, particularly in relation just general encoding/decoding infrastructure as well as Bandaid (where racks are shipped to ISPs to cache videos).
Android was another acquisition that prided itself in not being Google. This was very much fostered by Andy Rubin while he was still there. Obviously Google needed to write Android apps but I got the sense that it was always Google engineers who solved all the problems whereas Android just didn't care. They cared only about shipping Android. Fuchsia was an Android offshoot.
Docs and Maps were both acquisitions but they went fully Google3 and were different orgs but weren't seen as separate. The engineering director of Docs (Fuzzy) had, from what I can recall, a very positive reputation beyond Docs (now Drive).
Doubleclick was also an acqusition but went fully Google and you'll find a lot of people who don't even know it was an acquisition.
I don't know what org you worked in but they all vary. My own experience was that Infra orgs in comparison to Google were primitive and barely above just running random Docker-like (Tupperware) instances with a godawful variant of C++, probably started by someone who had done C++ at Google and had decided they really wanted mutable function parameters and exceptions for no particular reason.
The thing I really respected about FB product orgs generally was that really did ship things quickly. I used to joke that the smallest unit of time at Google was a quarter. God help you if you eneded another team (under a different VP) to do something. You'd have to spend a quarter arguing with them to get them to add it to their OKRs for the following quarter.
At FB the timeline for launching a new thing to a limited audience was measured in weeks. The biggest barrier usually was the weekly build cycle for the blue app. The release cycle for Web was S-tier and (IMHO) the people who worked on the infra for Web were generally god tier. This was another reason why IG doggedly sticking with Python just created problems.
There are many thigns you can criticize Meta for (eg the stupid crypto, the billions wasted on VR) but the Web Foundation and Ent teams were god tier and I'll die on that hill.
Anyway, even back then the ML teams and infra, not to put too fine a point on it, sucked (IMHO). Newsfeed was OK but the recommendations for a lot of things like videos just sucked. All of this was mainly because it all relied on daily offline jobs. And then Tiktok came along and showed everybody (including Youtube) just what a bad job they were doing at recommendations. And don't get me started on the IG Reels dumpster fire.
Oh ok, I thought of another one: Messenger. I knew some smart people on Messenger but overall the product and the infra were, again, a dumpster fire.
That's funny about your last line about Messenger. When I was at Facebook in 2020 and 2021 I thought the "Lightspeed" version of Messenger [1] was something of an engineering marvel. Specically the way it was centered around SQLite as the datastore, how the sever side could integrate with it, and how the UI was power by this reactive-style framework written in C. With lots of testing to keep binary size under control and things working reliably. And the DX was not bad for all these constraints. I think CG/SQL [2] is one of the few open source artifacts from this effort.
There were other reasonably nicely engineered things there, like some of the code-as-configuration things in PHP like the ORM and the structured logging. Where you just write and commit code and the databases and data warehouses just get set up automatically.
The choice to have a monolithic web stack at Facebook had some benefits. The latency from landing a diff to being in prod was a couple of hours, not matter what part of the site you touched. And given an arbitrary URL, there was a tool that used static analysis to tell you which file handled that request. Compared to Google where every damn thing is a micro-service with its own variable release schedule and different ways of doing request routing. Trying to figure out where a request goes is combination of dynamic tracing of the RPC trace behind it plus a bunch of grepping through the related code bases.
All that said, I was far away from doing real engineering at Facebook. I accidentally joined a growth team. Which meant our teams remit was churning out all kinds of experiments to see what we could to do to make our team's number go up. I did not find this type of work enjoyable and quit. So I have a narrow perspective.
Also to give these multinational corporations some benefit of the doubt, some of the way they operate is path-dependent. They have hundreds of millions of lines code. So they can get to a point where it makes sense to hire a bunch of compiler engineers to fork PHP and make the Hack language. Whereas a startup would almost certainly never create a new programming language to write their product. To take solutions of this context and apply them to the outside world might not make sense.
Thank you for the insight this actually answer a lot of questions or suspicious I had with Google. Surprised about Android and now Fuchsia makes sense.
I wonder if anyone has worked in Google / Facebook and Shopify and I wonder how they compare.
> I thought of another one: Messenger. I knew some smart people on Messenger but overall the product and the infra were, again, a dumpster fire.
That makes sense from a consumer point of view. I don't use FB much. In fact, I recently logged on a few times after a few years of a break. It's terrible. You can't even get conversation history anymore. You can talk to someone via the app on your phone, open Facebook in the browser on your computer and not see the messages there. Makes me wonder why is Meta still around?
If I were the CTO of any of these companies I would be working my butt off to be making an internal version of Claude. Let me explain my reasoning using Google as an example (disclaimer: Xoogler).
Google has a lot of systems to make a very large monorepo manageable so builds and code search don't take forever. The build system is Blaze (on which Bazel is based), which has a Pythonic syntax and was once Python but that hasn't been the case (AFAIK) for over a decade. This means you build a massive digraph of build artifacts. By "large" I mean somewhere between 100M and 1B vertices (guessing). Loading that became a significant problem for a build so there's heavy caching around that. There's also heavy caching around build artifacts (ie Forge).
So, part of the issue with every developer using Claude is that you have a ton of inefficiency becasue everybody has a significant context. And what is context really? It's not too dissimilar to the build graph and/or code search you already have.
So the infra I would be working on would be some kind of "global context" or "context cache". Now a lot of context changes when you do a local change but a lot doesn't. As an ordinary engineer, you aren't generally modifying /base. You're modifying leaf nodes or branches for very few leaf nodes.
The reasons I see to do this are:
1. Cost-savings by deduplication;
2. Speed if context is partially-cached;
3. You avoid issues of sending out your codes to third-parties. In the case of Google or Amazon, if they use Claude at all, they would probably only be using their own clouds so they avoid this. But Uber doesn't have that luxury;
4. You avoid any issues of people using your prompts for responses for training and leaking any potential sensitie information that way;
5. You can use off-peak resources for a lot of this work;
6. You can control resources within your own pervasive resource management (in the case of Google); and
7. You can more easily integrate into internal tooling.
I also think that expanding compute power is the biggest risk to Anthropic (and OpenAI). There's a vast difference between a model you need a cluster of NVidia's finest to run vs one you can run on a Macbook Pro. We aren't there yet on a Macbook Pro but it'll only be a few years we are.
The costs of a) selfhosting a >100B param LLM model b) scaling it to a full company and c) maintaining it are all significant risky investments that is even more expensive in the short term.
Those are generally the core reasons most SaaSes exists. Additionally, (a) is the biggest issue because there is no open-weights model that can match GPT 5.5/Opus 4.8.
There's more history than this. Disclaimer: Xoogler (2010-2017).
When I first started the environment you used depended entirely on language. In the C++ and Python space, there was the vim and emacs divide. With Java it was more complicated. Some still used vim/emacs but a lot of people used Eclipse.
Now Eclipse was a real problem at Google because of the source control system. Java IDEs are primarily built to import binaries, specifically jars. In the outside world, these dependencies are managed via Ant (very early days), Maven/Gradle or the like.
At Google there's a mono-repo (Perforce/Piper) and you check out parts of it locally and rely on the rest via a network connection (to SrcFS IIRC, it's been awhile). This was neat because you could edit a file locally and the dependencies would just recompile (via Blaze).
So for Eclipse a whole lot of initialization had to be done and the IDE would fall over. A lot. It had a team of ~10 working on it at one point. Then somebody did a 20% project called magicjar. Magicjar took a Perforce client and built all the dependencies as jars that could be imported directly without parsing the entire source tree (which was usually huge). This made it possible, even preferred, to use IntelliJ, which is what I did. Magicjar was great.
Other people actually made CLion work reasonably well with C++ too. That was nice. This was a much bigger undertaking with many more corner cases just given how C++ works (ie headers and templates).
So checking out a client was relatively heavyweight, even with a minimal local tree. And, if you worked on Google3, you had to do this a lot. You might need to do a config file change. This was the real starting point for Cider because it was way nicer to do config file changes with it.
Obviously I don't know where all this went from there. VS Studio as a Cider frontend? Ok, that was news to me. Engineers being unhappy when things change and when the slightest thing works differently is the least surprising thing I've ever heard.
Oh it's worth adding that in my time many people didn't use Perforce (P4) directly. They used somebody else's project, which was a Git frontend for it, called Git5. I believe it was already being deprecated while I was still there. But Git5 modelled a P4 change as a branch so you could play around with your Git commits locally and then squash them into a single P4 change. I actually liked this a lot.
One important piece of context that might make all these stories less confusing for non-googlers:
Code references are less important inside Google editors, because we have a code viewer tool inside the web browser.
Most people read, explore, follow references, and share permalinks to the view-only tool. It’s a lot better than viewing code in GitHub. It’s super fast, is connected to language servers and can actually trace referenced, and overall has a million little features optimized for reading code.
We also have a code reviewer tool, and a separate tool to run and view CI runs.
So what’s left for the editor? Syntax highlighting?
I would tend to view code, run tests and CI, and review in separate tools specialized for their specific use case. The code editor was just a place where I would type in my changes.
I’d imagine this workflow feels weird to people who learned in one-stop-shop IntelliJ and GitHub world. But I can’t emphasize how much better these other tools were compared to GitHib. So a code editor that also lets me read, review, and test code didn’t really matter for me when I had a collection of smaller tools specialized for each individual task.
To make this more concrete, the Chromium source code browser has a subset of the functionality of the internal Code Search tool. For example, you can left click on symbols to go to reference and right click to find all references:
Well, indexing and searching is kind of Google's thing.
It also helps that they're using a unified build system. They can instrument the compiler to get cross references from the build instead of trying to figure out out by parsing the text themselves.
They open sourced the tool to do it- https://kythe.io - but I think it would be a pain to make it work for anything like GitHub that supports arbitrary languages and build systems with untrusted code
This is a good example of large companies wouldn't send someone across the street to pick up $1M off the ground. If Google actually released that and a repo to public, they could take Githubs throne. But a few $B business isn't worthit for them.
It's not that the $2B business isn't interesting, it's competing with GitHub would be a major undertaking and the opportunity cost of doing that is probably more than $2B.
Brings back lots of memories. The only way i used that tool was back when Chrome stopped forever history near about the year 2014-2015 and limited users to 90 days of history.
I was trying to wrap around my head like reading the comments trying how to put it back in. Wrong strategy without serious review does more of harm.
More than a decade of this fiasco Google Chrome should bring it back forever history, now they dont have an excuse for this, what i hear (google Chrome now downloads an offline LLM without considering data charges or space requirements in edge deployed server,etc.), and it will help users themselves as now most browsers except Firefox or Safari are Chromium-Based and they too inherited this shiny features with not-so obvious limitations.
Brings back lots of memories. The only way i used that tool was back when Chrome stopped forever history near about the year 2014-2015 and limited users to 90 days of history.
I was trying to wrap around my head like reading the comments trying how to put it back in. Wrong strategy without serious review does more of harm.
More than a decade of this fiasco Google Chrome should bring it back forever history, now they dont have an excuse for this, what i hear (google Chrome now downloads an offline LLM without considering data charges or space requirements in edge deployed server,etc.), and it will help users themselves as now most browsers except Firefox or Safari are Chromium-Based and they too inherited this shiny features with not-so obvious limitations.
Fun read seeing even with all the tools in the world there is not enough soul or will to fix a very minor regression made by human error of judgement and other companies just accept that like their lives depend on it.
This...I noticed a real productivity increase when going from Cider/VSCode to JetBrains/IDEA/IntelliJ for Kotlin code editing. Having a "real" IDE was still a plus, if just for the better code completion.
AI has mostly changed the way I write code, I guess, so I rarely use JetBrains anymore, but a few years ago it was clearly a win to use a real IDE at least for Kotlin programming.
JetBrains has several niches it excels in. DataGrip is by far the most important tool in my toolbox, as it allows me to work with every database type imaginable in one place (Databricks, Postgres, MSSQL, Oracle, etc.).
The startup I'm at (ersc.io) is working in this space (version control more than the IDE side of things), because, in my opinion, there just plain isn't any.
FWIW your website is broken in both Firefox and Chrome. Huge rounded black boxes, social media icons larger than the screen, e.g. blog posts get 404 with an XML NoSuchKey error.
To be fair, code search still sucked for navigation comparing to IDE even in 2024. Even cider-v was rather so-so if you needed to navigate complex c++ code.
But I still remember days "edit in IDEA, debug in Eclipse"
Every large tech company of 80s, 90s & 2K (Google, eBay & all) have similar history when it comes to IDE, Source Control, Build Systems etc.,. This is not specific to Google.
I recall a couple of JetBrains staff visiting in 2008 to see IntelliJ's struggle with the size of checkout for AdWords. Soon after the reindexing front/center dialog moved from blocking and center of window, to a status line message that was non-bloking of edits at the bottom of the window. It may have been due anyway but was shown as problematic in the same moment.
You don't really need it anymore - CitC let you do views (mapping just part of the monorepo into your filesystem via FUSE) since about 2013, and then that functionality just got built into Piper. When I returned in 2020 you'd have a file at the top of your source tree that included all the relevant file mappings as well as any Blaze flags needed to build the project, and you could just point your IDE at that and it'd map in just what you need.
The history of Google's relationship to version control is even more interesting than editors - it went from CVS in 1998 to Perforce (P4) in 2000, then gcheckout and g4 in ~2006, then OverlayFS was invented in 2008, git5 came out in 2009, CitC obsoleted OverlayFS in ~2012, Piper built this all into the VCS in ~2013-2014, while I was gone from 2014-2020 apparently we got hg and jujutsu frameworks, and then when I got back in 2020 you'd just check out a .blazeproject from your IDE and everything would magically work. Many of these started as 20% projects (I used to have lunch with the guy who invented OverlayFS; interesting character and one of the best programmers I knew) and then got folded into the "official" way of doing things once grassroot adoption showed the execs that this was how people really wanted to work.
Haven't tried. But there are "IDEs" (Unity) that are really hostile to the idea that your project directory is not a fast local disk that they can both fill with garbage and use fsnotify on everything.
+1. CitC and it's relationship to version control (perforce/piper) is central to all development.
Let's say there were 100M file in the monorepo (an underestimate). You obviously never want to do a git clone of that. But what if clone and checkout were free? That's what CITC did. Creating a new workspace took less than 1 second and got you a FUSE filesystem that looked like it had everything in the repo. But nothing was actually downloaded until you opened files. And your local changes were also stored in the service. And this was available to the CI machines. See where I am going. CI did not have to clone the repo and apply your branch. CI just had your changes available. If you were just testing your 10 files there was no cost to having 100M other source files that were unrelated to your project.
This solves many problems that git worktrees attempt to address, but 10 years earlier, at less local disk cost, and more performant. I miss that.
When I joined in 2016, it CitC would make it look (and still does) like you had the entire monorepo on your local filesystem on your machine.
Git5 would copy some directories but builds would still fallback to files from the monorepo if you didn't track them. It was convenient for me since I could just grep and do fuzzy matching from my editor. Now I have to do some extra work to avoid grepping the entire monorepo. LLMs sometimes still try to grep the entire repo lol.
Now, you could use a perforace, mercurial, or jj interface and it works fine.
I started a touch before this in London. I recall before Blaze and git5 - every morning we had a ritual of checking out google3 and making sure we could get some sort of build working for the day so that we could then attempt to write some software on top of it. The builds in play were “Mach” and “quickie” or something like that. It was so painful we used to agree that we wouldn’t grab food or coffee or anything until we’d worked out what CL we should sync to for the day to do some work on.
Pair programming was very in vogue and I used to get in a little later than some which was a great excuse to just hop on someone else’s machine who’d already gone through that pain
Cider (and p4/g4c etc) was amazing when I left back in 2020, I loved it so much, and truly miss it. I rejoined Google last year, and they'd replaced it with a VSCode clone that truly was just a glorified text editor and most were all-in on mercurial as a piper/citc shim -- I was only there for 5 months before I decided not to stay, and I never managed to get Go type definition hints working.
That is not quite the right word. For Python, the headcount was moved from the Bay Area (the most expensive place in the world to hire software engineers) to Munich (the most expensive place in Germany to hire SWEs.), for cost saving reasons.
Most of the engineers making most of the tools being praised in this thread are in Germany, so I don't think that generalization quite holds.
Even if the best SWEs are better in the Bay area, there's also a lot more competition for them, so Google in Germany might be able to get top 1% there (and in neighboring countries) but Google in the Bay Area is probably having a tough time getting even top 10%.
That's a good point, and why I'm happy to see remote offices pop up in many locations. The problem is the top .1% which can live anywhere, is often a poor representation of the depth of talent density.
Similar to that IBM/Rational ClearCase, both are so unfriendly compared to subversion/cvs or git/mercurial that I always struggle to believe why someone would torture themselves using that. Probably admins love them because they allow some tooling to be added.
I worked at Google a long time ago now during the whole Google+ fiasco. One thing that was super controversial internally was the so-called Real Names policy. For those who don't know or remember, it was Vic Gundotra's idea that people should use their real identities. He kept using this weird example that he didn't want it filled with people named "Dog Turd". I don't know why.
So there was this mysterious black box that decided if your name was "real" or not. At first this didn't support pseudonyms or any kind of anonymity and that's actually really important for any social network. Think of someone seeking help coming to terms with their sexual orientation, gender identity, addiction, eating disorder or whatever. Or simply going against their family's religious wishes. I later worked at Facebook and one thing I'll give them credit for is Groups. FB Groups had an identity that actually couldn't be tied by anyone else to your profile or identity in any other group. That was a good product decision.
Anyway, if your name somehow failed the magic real names filter, your account got banned. Your entire Google account was banned and basically there was no recourse other than knowing someone who worked at the company or making a big enough fuss on Twitter.
Many people, myself included, criticized and protested this decision. You should at least segment Google products. There's absolutely no reason to ban your Gmail account because an automated system decided your Google+ account name wasn't "real". But that feedback was ignored and this was well before the public launch. And the public backlash proved this position correct (IMHO).
But the net effect was that I decided I can't use any other Google product. Let's say a system is launched to find offensive photos and there's a false positive on one of my images in Google Photos. Maybe it's just a hash collision with a known image. And then what? I lose my entire Gmail? Are you kidding me?
It's wild to me that this is still an issue ~15 years later. I think my stance actually isn't strict enough anymore. You probably shouldn't use Gmail at all. I should really find a paid email provider hosted entirely in Europe, preferably Switzerland or some other country with strong pro-user regulation.
So I have no idea if this Gemini story is true or not. I say that because 95% of the things on Reddit are completely made up. But it is plausible. I wouldn't be surprised if it's true. It means I wouldn't use Gemini at all if I used Gmail.
Story time. I used to work for Facebook (and Google) and lots of games were played around bugs.
At some point the leadership introduced an SLA for high then medium priority bugs. Why? because bugs would sit in queues for years. The result? Bugs would often get downgraded in priority at or close to the SLA. People even wrote automated rules to see if their bugs filed got downgraded to alert them.
Another trick was to throw it back to the user, usually after months, ostensibly to request information, to ask "is this still a problem?" or just adding "could not reproduce". Often you'd get no response. sometimes the person was no longer on the team or with the company. Or they just lost interest or didn't notice. Great, it's off your plate.
If you waited long enough, you could say it was "no longer relevant" because that version of the app or API had been deprecated. It's also a good reason to bounce it back with "is still this relevant?"
Probably the most Machiavellian trick I saw was to merge your bug with another one vaguely similar that you didn't own. Why? Because this was hard to unwind and not always obvious.
Anyone who runs a call center or customer line knows this: you want to throw it back at the customer because a certain percentage will give up. It's a bit like health insurance companies automatically sending a denial for a prior authorization: to make people give up.
I once submitted some clear bugs to a supermarket's app and I got a response asking me to call some 800 number and make a report. My bug report was a complete way to reproduce the issue. I knew what was going on. Somebody simply wanted to mark the issue as "resolved". I'm never going to do that.
I don't think you can trust engineering teams (or, worse, individuals) to "own" bugs. They're not going to want to do them. They need to be owned by a QA team or a program team that will collate similar bugs and verify something is actually fixed.
Google had their own versions of things. IIRC bugs had both a priority and s everity for some reason (they were the same 99% of the time) between 0 and 4. So a standard bug was p2/s2. p0/s0 was the most severe and meant a serious user-facing outage. People would often change a p2/s2 to p3/s3, which basically meant "I'm never going to do this and I will never look at it again".
I've basically given up on filing bug reports because I'm aware of all these games and getting someone to actually pay attention is incredibly difficult. So much of this comes down to stupid organizational-level metrics about bug resolution SLAs and policies.
> Google had their own versions of things. IIRC bugs had both a priority and s everity for some reason (they were the same 99% of the time) between 0 and 4. So a standard bug was p2/s2. p0/s0 was the most severe and meant a serious user-facing outage. People would often change a p2/s2 to p3/s3, which basically meant "I'm never going to do this and I will never look at it again".
Yeah, I've done that. I find it much more honest than automatically closing it as stale or asking the reporter to repeatedly verify it even if I'm not going to work on it. The record still exists that the bug is there. Maybe some day the world will change and I'll have time to work on it.
I'm sure the leadership who set SLAs on medium-priority bugs anticipated a lot of bugs would become low-priority. They forced triage; that's the point.
> People even wrote automated rules to see if their bugs filed got downgraded to alert them.
This part though is a sign people are using the "don't notify" box inappropriately, denying reporters/watchers the opportunity to speak up if they disagree about the downgrade.
> Google had their own versions of things. IIRC bugs had both a priority and severity for some reason (they were the same 99% of the time) between 0 and 4.
At the company I worked with (not Google, but a major one) this was the same. We used Salesforce, the "Lightning Experience" or whatever it was called [0]. Our version was likely customized for our company, but I think the idea was the same - one, I think the "priority", was for our eyes only, one was for the customer (the "severity"). If the customer was insistent on raising the severity, we'd put it as sev1, but the priority was what we actually thought it was. I was actually surprised that for the ~4 years I was there no one made the mistake of telling the customer the priority as a mistake, especially when a lot of people were sloppily copy-pasting text from Slack or other internal tools that sometimes referred to a case as either the severity or the priority.
Those were heavy customers with SLAs, though, not supermarket apps or anything like that.
What was sad was that our internal tools, no matter how badly written, with 90's UI and awful security practices, our tools were 50 times as fast as whatever Salesforce garbage we had to deal with. Of course, there was a lot of unneeded redundancy between the tools so the complexity didn't stay in the Salesforce tool. But somehow the internal tools written by someone 10 years ago, barely maintained, who had to still deal with complex databases of who-what-when-how, felt like you had the DB locally on a supercomputer while SF felt like you were actually asking a very overworked person to manually give you your query right on each click. I'm exaggerating, but just by a bit.
[0] That name was funny because it was slow as shit. Each click took 5 to 20 seconds to update the view. I wonder what the non-Lightning version was.
> IIRC bugs had both a priority and s everity for some reason (they were the same 99% of the time) between 0 and 4. So a standard bug was p2/s2. p0/s0 was the most severe and meant a serious user-facing outage
I've seen this at a couple places... I think it's supposed to help model things like if something is totally down, that's an S0... But if it's the site for the Olympics and it's a year with no Olympics, it's not a P0.
Personally, that kind of detail doesn't seem to matter to me, and it's hard to get people to agree to standards about it, so the data quality isn't likely to be good, so it can't be used for reporting. A single priority value is probably more useful. Priority helps responsible parties decide what issue to fix first, and helps reporters guess when their issue might be addressed.
> People would often change a p2/s2 to p3/s3, which basically meant "I'm never going to do this and I will never look at it again".
I learned this behavior because closing with wontfix would upset people who filed issues for things that I understand, but am not going to change. I'm done with it, but you're going to reopen it if I close it, so whatever, I'll leave it open and ignore it. Stalebot is terrible, but it will accept responsibility for closing these kinds of things.
> I don't think you can trust engineering teams (or, worse, individuals) to "own" bugs. They're not going to want to do them.
I will disagree there. The engineers often want to fix the bugs. Management is telling them they need it's all hands on deck for (Insert company goal here. Probably AI right now).
Followed by management also telling them they have too many bugs, of course. In a condescending tone.
So there's some survivor bias here but it's generally not bad advice. You should be focusing on outcomes like improving SLAs, top line metrics and so on. You should be solving user and business problems. That's all good advice. But still this article presumes a lot.
In my experience, managers will naturally partition their reports into three buckets: their stars, their problems and their worker bees. The worker bees tend to be ignored. They're doing fine. They're getting on with whatever they've been told to do or possibly what they've found to do. They're not going to create any problems. The problems are the underperformers. These are people who create problems and/or are at risk of getting a subpar performance rating.
Now there are lots of reasons that someone can be a problem. I tend to believe that any problem just hasn't found the right fit yet and, until proven otherwise, problems are a failure in management. That tends to be a minority view in practice. It's more common to simply throw people in the deep end and sink or swim because that takes much less overhead. You will see this as teams who have a lot of churn but only in part of the team. In particularly toxic environments, savvy managers will game the system by having a sacrificial anode position. They hire someone to take the bad rating they have to give to protect the rest of the team.
And then there are the stars. These are the people you expect to grow and be promoted. More often than not however they are chosen rather than demonstrating their potential. I've seen someone shine when their director is actively trying to sabotage them but that's rare.
Your stars will get the better projects. Your problems will get the worse ones. If a given project is a success or not will largely come down to perception not reality.
The point I'm getting to is that despite all the process put around this at large companies like performance ratings, feedback, calibration, promo committees, etc the majority of all this is vibes based.
So back to the "take my job" advice. If someone is viewed as a star, that's great advice. For anyone else, you might get negative feedback about not doing your actual job, not being a team player and so on. I've seen it happen a million times.
And here's the dirty little secret of it all: this is where the racism, sexism and ableism sneaks in. It's usually not that direct but Stanford grads (as just one example) will tend to vibe with other Stanford grads. They have common experience, probably common professors and so on. Same for MIT. Or CMU. Or UW. Or Waterloo. And so on.
So all of the biases that go into the selection process for those institutions will bleed into the tech space.
And this kind of environment is much worse for anyone on the spectrum because allistic people will be inclined to dislike from the start for no reason and that's going to hurt how they're viewed (ie as a star, a worker bee or a problem) and their performance ratings.
Because all of this is ultimately just a popularity contest with very few exceptions. I've seen multiple people finagle their way to Senior STaff SWE on just vibes.
And all of this gets worse since the tech sector has joined Corporate America in being in permanet layoff mode. The Welchian "up or out" philosophy has taken hold in Big Tech where there are quotas of 5-10% of the workforce have to get subpar ratings every year and that tends to kill their careers at that company. This turns the entire workplace even more into an exercise in social engineering.
Yeah the only solution to avoid this is to find a company where building and selling a product actually matters. In large companies it’s too easy to fudge the connection between individual contributions and financial impact.
If you’re not looking to become a founder, companies right around 100 employees is the sweet spot in my (very limited) experience.
> 2. Being right is cheap. Getting to right together is the real work
> 6. Your code doesn’t advocate for you. People do
> 14. If you win every debate, you’re probably accumulating silent resistance
The common thread here is that in large organizations, your impact is largely measured by how much you're liked. It's completely vibes-based. Stack ranking (which Google used to have; not sure if it still does) just codifies popularity.
What's the issue with that? People who are autistic tend to do really badly through no fault of their own. These systems are basically a selection filter for allistic people.
This comes up in PSC ("perf" at Meta, "calibration" elsewhere) where the exact same set of facts can be constructed as a win or a loss and the only difference is vibes. I've seen this time and time again.
In one case I saw a team of 6 go away and do nothing for 6 months then come back and shut down. If they're liked, "we learned a lot". If they're not, "they had no impact".
Years ago Google studied the elements of a successful team and a key element was psychological safety. This [1] seems related but more recent. This was originally done 10-15 years ago. I agree with that. The problem? Permanent layoffs culture, designed entirely to suppress wages, kills pyschological safety and turns survival into a game of being liked and manufacturing impact.
> 18. Most performance wins come from removing work, not adding cleverness
One thing I really appreciated about Google was that it has a very strict style guide and the subset of C++ in particular that you can use is (was?) very limited. At the time, this included "no exceptions", no mutable function arguments and adding templtes had an extremely high bar to be allowed.
Why? To avoid arguments about style issues. That's huge. But also because C++ in particular seemed to attract people who were in love with thier own cleverness. I've seem some horrific uses of templates (not at Google) that made code incredibly difficult to test for very little gain.
> 9. Most “slow” teams are actually misaligned teams
I think this is the most important point but I would generalize it and restate it as: most problems are organizational problems.
At Meta, for example, product teams were incentivized to ship and their impact was measured in metric bumps. But there was no incentive to support what you've already shipped beyond it not blowing up. So in many teams there was a fire and forget approach to filing a bug and forgetting about it, to the point where it became a company priority to have SLAs on old bugs, which caused the inevitable: people just downgrading bug priorities to avoid SLAs.
That's an organizational problem where the participants have figured out that shiping is the only thing they get rewarded for. Things like documentation, code quality and bug fixes were paid lip service to only.
The mistake was not having nullability be expressed in the type system.
At Facebook I used their PHP fork Hack a lot and Hack has a really expressive type system where PHP does not. You can express nullability of a type and it defaults to a type being non-nullable, which is the correct default. The type checker was aware of changes too, so:
function foo(?A $a): void {
$a->bar(); // compile error, $a could be null
if ($a is null) {
return;
}
$a->bar(); // not a compiler error because $a is now A not ?A
if ($a is ChildOfA) {
$a->childBar(); // not an error, in this scope $a is ChildOfA
}
}
Now Hack like Java used type erasure so you could force a null into something non-nullable if you really wanted to but, in practice, this almost never happened. A far bigger problem was dealing with legacy code that was converted with a tool and returned or used the type "mixed", which could be literally anything.
The real problem with Java in particular is you'd end up chaining calls then get the dreaded NullPointerException and have no idea from the error or the logs what was broken from:
a.b.c.d();
I'm fine with things like Option/Maybe types but to me they solve different problems. They're a way of expressing that you don't want to specify a value or that a value is missing and that's different to something being null (IMHO).
Yeah in Typescript I rarely run into null deference errors at runtime either. It can happen if you unsafely cast the type of values coming into your code, but if you runtime validate input at your application boundary it’s very unlikely
Now looking back at a lot of other languages that don’t express nullability, it’s like, what were they thinking? How did I not wish for nullability in type declarations in all my years of dealing with NullPointerExceptions?
> The real problem with Java in particular is you'd end up chaining calls ... and have no idea from the error or the logs what was broken from: a.b.c.d();
That’s been solved since Java 14. (5 years ago) Now the error will tell you exactly what was null.
And “soon” Java will have built in support for expressing nullability in the type system. Though with existing tools like NullAway it’s already (in my opinion) a solved problem.
Exactly, null is not evil it itself – the fact that it's not represented in type system is.
Type system where nullability can be expressed, you have refinement so you can map "null | T" to "T" with conditionals and sugar like optional chaining and nullish coalescing is all that's needed.
In PHP land, for some years now that code would not pass CI/CD checks and IDEs show red squiggles. Provided they use any popular static analysis tools like PHPStan, Psalm and I believe SonarQube would also flag it.
As an early user of SO [1], I feel reasonably qualified to discuss this issue. Note that I barely posted after 2011 or so so I can't really speak to the current state.
But what I can say is that even back in 2010 it was obvious to me that moderation was a problem, specifically a cultural problem. I'm really talking about the rise of the administrative/bureaucratic class that, if left unchecked, can become absolute poison.
I'm constantly reminded of the Leonard Nimoy voiced line from Civ4: "the bureaucracy is expanding to meet the needs of the expanding bureaucracy". That sums it up exactly. There is a certain type of person who doesn't become a creator of content but rather a moderator of content. These are people who end up as Reddit mods, for example.
Rules and standards are good up to a point but some people forget that those rules and standards serve a purpose and should never become a goal unto themselves. So if the moderators run wild, they'll start creating work for themselves and having debates about what's a repeated question, how questions and answers should be structured, etc.
This manifested as the war of "closed, non-constructive" on SO. Some really good questions were killed this way because the moderators decided on their own that a question had to have a provable answer to avoid flame wars. And this goes back to the rules and standards being a tool not a goal. My stance was (and is) that shouldn't we solve flame wars when they happen rather than going around and "solving" imaginary problems?
I lost that battle. You can argue taht questions like "should I use Javascript or Typescript?" don't belong on SO (as the moderators did). My position was that even though there's no definite answer, somebody can give you a list of strengths and weaknesses and things to consider.
Even something that does have a definite answer like "how do I efficiently code a factorial function?" has multiple but different defensible answers. Even in one language you can have multiple implementations that might, say, be compile-time or runtime.
Another commenter here talked about finding the nearest point on an ellipse and came up with a method they're proud of where there are other methods that would also do the job.
Anyway, I'd occasionally login and see a constant churn on my answers from moderators doing pointless busywork as this month they'd decided something needed to be capitalized or not capitalized.
A perfect example of this kind of thing is Bryan Henderson's war on "comprised of" on Wikipedia [2].
Anyway, I think the core issue of SO was that there was a lot of low-hanging fruit and I got a lot of accepted answers on questions that could never be asked today. You'll also read many anecdotes about people having a negative experience asking questions on SO in later years where their question was immediately closed as, say, a duplicate when the question wasn't a duplicate. The moderator just didn't understand the difference. That sort of thing.
But any mature site ultimately ends with an impossible barrier to entry as newcomers don't know all the cultural rules that have been put in place and they tend to have a negative experience as they get yelled at for not knowing that Rule 11.6.2.7 forbids the kind of question they asked.
> This manifested as the war of "closed, non-constructive" on SO. Some really good questions were killed this way because the moderators decided on their own that a question had to have a provable answer to avoid flame wars.
It's literally a Q&A site. Questions need actual answers, not just opinions or "this worked for me".
> This manifested as the war of "closed, non-constructive" on SO. Some really good questions were killed this way because the moderators decided on their own that a question had to have a provable answer to avoid flame wars.
Please point at some of these "really good" questions, if you saved any links. (I have privileges to see deleted questions; deletion is normally soft unless there's a legal requirement or something.) I'll be happy to explain why they are not actually what the site wanted and not compatible with the site's goals.
> I lost that battle. You can argue taht questions like "should I use Javascript or Typescript?" don't belong on SO (as the moderators did). My position was that even though there's no definite answer, somebody can give you a list of strengths and weaknesses and things to consider.
> What kind of questions are appropriate? Well, thanks to the tagging system, we can be rather broad with that. As long as questions are appropriately tagged, I think it’s okay to be off topic as long as what you’re asking about is of interest to people who make software. But it does have to be a question. Stack Overflow isn’t a good place for imponderables, or public service announcements, or vague complaints, or storytelling.
> Stackoverflow is sort of like the anti-experts-exchange (minus the nausea-inducing sleaze and quasi-legal search engine gaming) meets wikipedia meets programming reddit. It is by programmers, for programmers, with the ultimate intent of collectively increasing the sum total of good programming knowledge in the world. No matter what programming language you use, or what operating system you call home. Better programming is our goal.
Questions that are fun and slightly outside of the intended domain of the site are manageable ... if there is sufficient moderation to keep those types of questions from sucking up all available resources.
That was the first failing of NotProgrammingRelated.StackExchange ... later Programming.StackExchange ... later SoftwareEngineering.StackExchange.
The fun things, while they were fun took way more moderation resources than was available. People would ask a fun question, get a good bit of rep - but then not help in curating those questions. "What is your favorite book" would get countless answers... and then people would keep posting the same answers rather than reading all of them themselves and voting to cause the "good" content to bubble up to the top.
Stack Overflow kind of had this at its start... but over time the "what is acceptable moderation" was curtailed more and more - especially in the face of more and more questions that should be closed.
While fun questions are fun... the "I have 30 minutes free before my next meeting want to help someone and see a good question" is something that became increasingly difficult. The "Keep all the questions" ideal made that harder and so fewer and fewer of the - lets call them "atwoodians" remained. From where I sit, that change in corporate policy was completely solidified when Jeff left.
As moderation and curation restricted (changing the close reasons to more and more specific things - "it's not on that list, so you can't close it") meant that the content that was not as well thought out but did match the rules became more and more prevalent and overwhelmed the ability for the "spolskyites" to close since so many of the atwoodians have left.
What remained where shells of rules that were the "truce" in the tension between the atwoodians and spolskyites and a few people trying to fight the oncoming tide of poorly asked questions with insufficient and neglected tooling.
As the tide of questions went out and corporate realized that there was necessary moderation that wasn't happening because of the higher standards from the earlier days they tried to make it easier. The golden hammer of duplication was a powerful one - though misused in many cases. The "this question closes now because its poorly asked and similar to that other canonical one that works through the issue" was far easier than "close as {something}" that requires another four people to take note of it before the question gets an answer from the Fastest Gun in the West. Later the number of people needed was changed from needing five people to three, but by then there was tide was in retreat.
Corporate, seeing things there were fewer questions being asked measured this as engagement - and has tried things to increase engagement rather than good questions. However, those "let's increase engagement" efforts were also done with even more of a moderation burden upon the community without the tooling to fix the problems or help the diminishing number of people who were participating in moderating and curating the content of the site.
> As moderation and curation restricted (changing the close reasons to more and more specific things - "it's not on that list, so you can't close it") meant that the content that was not as well thought out but did match the rules became more and more prevalent and overwhelmed the ability for the "spolskyites" to close since so many of the atwoodians have left.
Just to make sure: I always got the impression that Atwood was the one who wanted to keep things strictly on mission and Spolsky was the one more interested in growing a community. Yes? I do get the impression that there was a serious ideological conflict there; between the "library of detailed, high-quality answers" and the, well, "to every question" (without a proper understanding of what should count as a distinct, useful question that can have a high-quality answer). But also, the reputation gamification was incredibly poorly thought out for the "library" goal (https://meta.stackexchange.com/questions/387356/the-stack-ex...). And I suspect they both shared blame in that.
A lot of it was also ignored for too long because of the assumption that a) the site would just die if it clamped down on everything from the start; b) the site would naturally attract experts with good taste in questions (including maybe even the ability to pose good https://en.wikipedia.org/wiki/Dorothy_Dixer questions) before the beginners ever cleared the barrier of trying to phrase a proper question instead of using a forum.
(Nowadays, there are still small forums all over the place. And many of them try to maintain some standards for the OP. And they're all plagued with neophytes who try to use the forum as if it were a chat room. The old adage about foolproofing rings true.)
Around 2014 is when the conflict really seems to have boiled over (as new question volume was peaking). Notably, that also seems to be when the dupe-hammer was introduced (https://meta.stackoverflow.com/questions/254589).
Joel was indeed more community minded - though part of that community mindedness was also more expectations of community moderation than what the tooling was able to scale for.
And yes, they both were to blame for gamification - though part of that was the Web 2.0 ideals of the time and the hook to keep a person coming back to it. It was part of the question that was to be answered "how do you separate the core group from the general participants on a site?" ... and that brings me to "people need to read A Group Is Its Own Worst Enemy" ( https://news.ycombinator.com/item?id=23723205 ) to understand how it shaped Stack Overflow.
Atwood: Maybe. But the cool thing about this is this is not just me, because that would be boring. It is actually me and Clay Shirky. You know, Clay Shirky is one of my heroes.
Spolsky: Oh...
Atwood: Yeah I know, it's awesome. So we get to talk about like building communities online and I get to talk about StackOverflow, you know, and all the lessons we've learned and, get to present with Clay. Obviously he's an expert so. That's one of the people that I have emailed actually, because I thought that would be good, because he is from New-York city as well. So we could A) show him the site and B) talk about the thing we are going to do together in March, because he needs to see the site to have some context. I mean I did meet him and talk to him about this earlier a few months ago, I think I mentioned it on the podcasts. But that was before we had sort of even going to beta, so there's really not a lot to show him. But I would love to show him in person. So we'll see if I'll hear back from him, I do not know.
2014 sounds about right for when it peaked... it was also when a lot of things hit the fan one after another. General stress, the decline of community moderation. The dup hammer was a way to try to reduce the amount of close votes needed - but in doing so it became "everything is a nail" when the dup hammer. It was used to close poor questions as dups of other questions ... and rather than making it easier to close questions that didn't fit well, corporate allowed the "everything is a dup" problem to fester.
How much traffic do the questions that get duped to something bring? Especially the (currently) 410 questions linked to the Java NPE question.
That question now has 10,356 questions linked to it... and that's part of the "why search quality is going down" - because poor questions were getting linked and not deleted. Search went downhill, dupe hammer was over used because regular close votes took too long because community moderation was going down, which in turn caused people to be grumpy about "closed as dup" rather than "your question looks like it is about X, but lacks an MCVE to be able to verify that... so close it as a dup of X rather than needing 5 votes to get an MCVE close.. which would have been more helpful in guiding a user - but would mean people would start doing FGITW to answer it maybe and you'd get it as a dup of something else instead."
Having duplicates should make the search better, by pointing people who phrase the same problem in different ways to the same place. But low-quality questions often don't produce something searchable for others, and they cover topics relevant to people who lack search skills.
Dunno why you are being downvoted - there is a certain type of person who contributes virtually nothing on Wikipedia except peripheral things like categories. BrownHairedGirl was the most toxic person in Wikipedia but she was lauded by her minions - and yet she did virtually no content creation whatsoever. Yet made millions of edits!
For anyone unfamiliar, modern video codecs tend to break up a video into a group of pictures (GOP) and you have a mix of different frame types. Some of these are the entire image. Most are just differences. It gets more complex as the video codecs allow you to pull objects from previous and future frames. But I digress.
The point is one of the things you have to decide is how often you send a complete image. Typically that's every 1-2 seconds. It may be more often with scene changes. The bigger the minimum gap in full images, the lower the bandwidth. The downside? The longer it takes to start playing a channel. Also, it makes network interruptions worse as you may have distorted the image and sound for up to 2 seconds instead of 1.
You also have to make a choice between CBR (constant bit rate) and VBR (variable bit rate). CBR is often preferred for mass distribution because it doesn't overload what are usually lower powered CPUs but that will also lead to a delay in channel changes as a full image may take longer than 1/24 or 1/30 of a second to actually send.
And then when you get into sending video over a network you have to worry about some Russian dolls of various standards. The video has a format. So does the audio. Those are in a container format (eg MKV). And then you have a transport format. For Fiber this was (and probably still is but I'm guessing) MPEG-TS (Transport Streams), which is a weird format that goes all the way back to ATM. Basically 7 packets were put into a single multicast IPv4 packet. Why 7? Because it's the most you could fit while staying under an MTU of 1500. As anyone who has dealt with IP, things just stop working or get more difficult with large MTUs.
Now on top of all of that and minimum 2-3 second channel switches you need to deal with multicast channel subscription delays. I don't know if this was changed with IPv6 but IPv4 multicast just wasn't designed with low latency subscription changes in mind. This goes to an underlying philosophy TCP/IP had in its inception of at-most once delivery. In the modern world, we've come to realize that's wrong for streaming content and we want at-least once delivery instead.
But it gets worse than that. When playing video you generally want to have a reasonable buffer. This way you have time to recover if you have some lost or damaged packets. Multicast doesn't give you a way of dealing with that so you need to build a whole different set of infrastructure for that.
Video playback in general is really difficult to get right, reliably. As an eample, my Chrome for reasons I haven't been able to ascertain, stutters every second when playing certain csources. Firefox doesn't. I suspect it's something to do with the aofrementioned full image frame every second. If Google can't get that right, is it any wonder that so many problems exist?
Circling back, multicast is a really awkward fit for something like this and it's not going to be a great user experience but, somewhat surprisingly, the alternatives still have issues.
Take for example DASH. This is a more modern method of video transport except it's kind of weird because you have an XML manifest file. This works quite well for VODs. There are space-saving variants that allow for essentially templated frame names rather than listing each frame and stream idnividually. But when you get to streaming live video, it gets rather awkward because the client may end up requesting a segment that doesn't exist yet. What should you do? Do you block the HTTP request and wait for it? Do you return a 404?
I guess this is a really long-winded way of saying I'd never do multicast distribution unless I had to. Or I was just playing around. It would be interesting to see what devices just absolutely choke when trying to consume multicast IPv6 video and the various container and stream formats that may entail.