Wednesday, February 1, 2023
HomeEconomicsTwo Methods of a Blue Fowl: What's Actually Happening at Twitter?

Two Methods of a Blue Fowl: What’s Actually Happening at Twitter?

Twitter? Twitter? Twitter? Twitter?” “Um, Twitter’s sick. My greatest buddy’s sister’s boyfriend’s brother’s girlfriend heard from this man who is aware of this child who’s going with the woman who noticed Twitter go out at 31 Flavors final night time. I assume it’s fairly critical.” That’s principally the place we’re with the protection. In the meantime: “‘Twitter Is Useless,’ 300 Million Individuals Publish On Twitter.”

I like to think about myself as technically literate, moreso again within the day than I’m now. Because of my Internet 1.0 heritage, I can arrange and run an internet site, together with the backend. Nonetheless, the methods that Twitter engineers are (or have been) sustaining at scale are unimaginably complicated to me. These methods — their design and implementation, who runs them, what they ship to customers — are on the coronary heart of the Twitter story. Sadly, the primary two points of Twitter are opaque simply now, so discovering out what’s actually occurring may be very tough (and on this story, Twitter the platform isn’t serving to very a lot). So what’s actually occurring?

Probably the most fundamental query: Is Twitter down? The reply as of this writing (12:03PM EDT, November 21, 2022) isn’t any. Twitter isn’t down. Twitter’s APi web page says all methods are inexperienced:

Will Twitter go down? Down for Everybody or Simply Me? says it has, so likely it’s going to:

Curiously, the final 5 outages are all since Musk’s ascension to CEO, however that’s way back to the info I can discover goes (and Twitter’s file of being up has been no means good. I can say, nonetheless, that as a particularly on-line one that virtually lives on Twitter, that I didn’t really feel a factor).

On this put up, I’ll have a look at two points of Twitter. First, I’ll have a look at Twitter’s software program, and present why the World Cup will stress it. Then, I’ll have a look at the gamers within the ongoing Twitter drama, and the position of every in Twitter’s present drama. I can even add an Appendix on Mastodon, Twitter’s open supply competitor.

Twitter Software program

The Twitter software program handles a lot of knowledge. From the Twitter Engineering Weblog, “Subsequent technology information insights utilizing pure language queries“:

At Twitter, we course of roughly 400 billion occasions in actual time and generate petabyte (PB)-scale information every single day.

That’s lots. Within the face of this information circulate, the important technical problem to maintaining Twitter up — the baseline requirement for Twitter creating worth — is spikes. From the identical weblog, “When seconds actually do matter“:

On Twitter, the true world occurs in sub-minutely intervals and exercise on the platform follows. Whether or not it’s a retirement announcement from a member of a boy band, a authorities coup, a world vast soccer event objective, or just the beginning of a brand new yr, site visitors patterns change rapidly and unpredictably as these occasions unfold. These are the moments when the minutely mannequin turns into inadequate. Let’s think about {that a} movie star couple publicizes the beginning of a kid at two seconds after the closest minute — for this instance, let’s say 2:00:02. With this announcement, a spike of site visitors causes undue stress on one of many many micro providers[1] that powers Twitter inflicting service errors to some customers.

The classical instance of this form of spike is the load on sewage methods when one million bogs concurrently flush throughout a industrial break for the Superbowl. For Twitter, the 28 days of the World Cup pose an equal problem. From the New York Occasions:

[The World Cup] is predicted to carry a deluge of site visitors to Twitter, which is the world’s fourth most visited web site, in response to Similarweb, a digital intelligence platform that tracks internet site visitors. Twitter will get 6.9 billion visits every month, barely greater than Instagram’s 6.4 billion, although far fewer than Google, YouTube or Fb, in response to Similarweb estimates.

Previously, Twitter has a “Command Heart” to deal with the spikes, however many have left. From The Verge:

A number of members of Twitter’s “Command Heart” staff, a gaggle of engineers that’s on name 24/7 and acts because the clearing home for issues internally, additionally tweeted about their departures. “In the event that they go down, there isn’t any one to name when shit breaks,” mentioned an individual acquainted with how the staff operates.

“An individual.” Many of the protection on inside Twitter points is single-sourced. One other single supply from the Guardian:

Twitter stands a 50% likelihood of a significant outage that might take the location offline through the World Cup, in response to a lately departed worker with information of how the corporate responds to large-scale occasions.

The previous worker, who was granted anonymity due to the sensitivity of what was mentioned, has information of the workings of Twitter Command Centre, the platform’s staff of troubleshooters who monitor the location for points comparable to site visitors spikes and information centre outages.

Between the dearth of preparations and the dearth of staffing, I believe it’s going to be a tough World Cup for Twitter,” mentioned the previous worker.

He recommended that an incident of some variety – comparable to a service responding slowly or incorrectly – is sort of a certainty through the 29-day [sic] competitors in Qatar, estimating a 90% chance of one thing going incorrect that customers would see.

Effectively, we’ll simply have to attend and see. 28 days will likely be fairly a stress check.

From dealing with spikes, we flip to Twitter’s fundamental information construction, the graph (or “social graph”). Right here’s a quite simple diagram of a really small graph:

Extra technically, as soon as once more from Twitter’s weblog, “Graph ML at Twitter.” The circles within the picture above are “nodes.” The traces between them are “edges”[2]:

Graphs are mathematical abstractions of complicated methods of relations and interactions. A graph represents a community of objects (referred to as nodes or vertices) with pairwise connections (edges). Graphs are ubiquitously utilized in fields as numerous as biology, quantum chemistry, and high-energy physics’. . Public conversations occurring on our platform sometimes generate tons of of tens of millions of Tweets and Retweets every single day. This makes Twitter maybe one of many largest producers of graph-structured information on the earth, second maybe solely to the Giant Hadron Collider.

“Second maybe solely to the Giant Hadron Collider.” Oof! Since Twitter as a enterprise exists to monetize relationships between customers as expressed via replies, Retweets, and Likes, the representational energy of the Twitter graph is of the primary significance. Little doubt that’s why Musk could enhance it. From Harvard Enterprise College’s Andy Wu:

Doing that form of swap may be very, very harmful. So we’re going to have to attend and see what’s going to occur right here. My sense is that they’re prioritizing the interior core of the know-how of Twitter, the social graph.

So, what’s actually occurring with Twitter? Within the brief time period, an infinite stress check of the software program: The World Cup. In the long run, enhancing Twitter’s social graph.

Twitter Gamers

However what’s actually occurring? Let’s have a look at the gamers, and the struggles between them. The gamers fall into to buckets: The oligarchs, and the Skilled Managerial Class (PMC). Let’s have a look at every in flip.

First, the oligarchs. On this part, the set of all oligarchs has just one member: Musk. Software program luminary Grady Booch writes within the unrolled Twitter thread: “Twitter 2.0: a contemporary story about methods engineering“:

Take a legacy group that has through the years constructed a not good however moderately stable web-centric system at elastic world scale.

Let that engineering staff function below the demonstrably, poor management of @jack and the distinct lack of grownup supervision by Twitter’s board for too a few years.

Usher in a reluctant CEO with the charisma of Mussolini, the bluster of Trump, and the delicate nature of a bull in a china store.

Have that CEO finance the take care of a debt that will stun a staff of ox, a debt that needs to be serviced on prime of that group’s already destructive money circulate.

On day one and for a lot of extra days that observe, have that CEO alienate his newly-acquired customers, scare away his few advertisers, and decimate the event staff.

Have that CEO demand strict allegiance to his capricious edicts, institute unsustainable engineering processes, and tear aside the present infrastructure by ripping out components he barely understands.

So right here’s the factor:

Making a Twitter 2.0 is just partly a software program engineering downside.

It’s a methods engineering downside that requires a agency and regular hand on all the opposite shifting components such because the enterprise mannequin, understanding the wants of its prospects and different stakeholders, defining a compelling imaginative and prescient for the long run.


Elon is approaching Twitter 2.0 as if it’s only a “hardcore software program engineering” downside.

That’s the place he’s gone off the rails.

Twitter 2.0 is actually not rocket science, however Elon is treating it like it’s. His give attention to the hardcore facet of it’s comprehensible but misguided, and represents a harmful distraction and alternative price.

I don’t know how this can evolve within the coming weeks.

However I do know that it will likely be uncomfortable to observe, because the lives of Twitter’s employees are upended and as this necessary world useful resource is trashed.

That is nearly as good an outline of Musk’s scenario as I’ve seen, and from a Grasp of the Nice Runes, too. (In Musk’s protection, he’s clearly fascinated about the enterprise mannequin, too; introducing funds to creators — Twitter has many, many artists, writers, performers, and many others. — can be a significant change to the enterprise mannequin, and a welcome one, too.)

Now, the PMC. For individuals who got here in late, the definition from Barbara and John Ehrenreich (1977):

Salaried psychological staff who don’t personal the technique of manufacturing and whose main perform within the social division of labor could also be described broadly because the replica of capitalist tradition and capitalist class relations.

Actually programmers; definitely managers. First, the managers. Matt Stoller writes:

Whereas I’m no fan of Musk, I do know that Twitter was horribly managed earlier than his takeover.

Grady Booch (above) famous the “demonstrably, poor management of @jack and the distinct lack of grownup supervision by Twitter’s board.” And from HBS’s Andy Wu:

Twitter was really in very unhealthy form and didn’t fairly have a future anyway. When it comes to actually robust issues, [Musk] is the form of CEO you in all probability have to check out. Twitter is definitely a really, very tough enterprise problem that no one else has been capable of clear up. So at this level, we would have to, like, swing the automobile round and see what occurs.

This can be a firm that couldn’t ship an Edit button to customers for the whole interval of its existence, despite the fact that customers have been clamoring for it. (Mastodon, Twitter’s open supply competitor, has an edit button.)

Now the professionals. These could be divided into employees (inside Twitter) and the press (outdoors, reporting on it). So far as the employees goes, others echo Gooch’s grudging reward (“not good however moderately stable). Andy Wu as soon as extra:

The individuals who constructed the Twitter product undoubtedly constructed a reasonably resilient infrastructure to outlive via all of this. The technical half can function with out numerous the folks.

An instance of such resilience: swish degradation (like when an internet web page hundreds a barely formatted model of itself, however nonetheless offers you one thing to learn:

The employees did some good engineering, there. (Different examples of the form of unsung enginering it takes to maintain a really giant system up and operating right here, right here, and right here.) In fact, from the oligarch’s perspective, none of this interprets immediately into headcount[3],[4]. Musk could determine he can rent again what he wants:

Right here, for instance, is a really comfortable camper who stayed:

Let’s simply hope the Peter Precept doesn’t apply right here.

And now the second set of execs, the press. The important thing level right here is that the press have executed what old-school journalists ought to by no means do: They’ve turn into a part of the story by doing little or no reporting, in “print” or on Twitter, as a substitute participating in an infinite dogpile of Elon Musk. (I discover the press’s pearl-clutching over a set of software program professionals shifting on to inexperienced fields and pastures new, and their silent indifference to, say, railroad employee or nurses, actually galling. I additionally discover the implication that Musk, and I suppose Trump, are the one billionaires worthy of censure galling as effectively.) For now, I’ll merely be aware the financial incentives. From the final sentence of this Occasions story:

Mr. Musk has more and more downplayed the position of conventional media over the previous few months, citing Twitter as probably the greatest platforms for the rise in “citizen journalism,” as he put it.

Oh. Can’t have that. Arthur wouldn’t prefer it.


As normal, Atrios is wise. “The Fowl App Will Be Gone And We Will All Be Free“:

Twitter had two principal contributions to the world (effectively, the US, anyway): It killed the affect of Drudge….

Fairly proper. I keep in mind when Drudge drove the information cycle. Having Twitter drive the information cycle, for all its many faults, is a gigantic enchancment.

… and instantly a bunch of immensely self-important folks have been confronted with tons of of individuals, typically deservedly, calling them shitheads every single day.

By calling them shitheads, I imply typically calling them shitheads backed up with good arguments. And numerous these arguments have been from folks of colour and different “riff raff” comparable to individuals who went to lesser colleges than Harvard.

Trivially, Twitter permits me to name Rochelle Walensky a eugenicist to her face (or her intern’s face), and have others see it. To me, that’s what’s actually occurring at Twitter. I’m very doubtful that there’s one other platform that allows that. Non-trivially, Twitter allowed aerosol scientists — a lot of whom are from [shudder] state colleges — to name out the droplet dogmatists efficiently (“backed up with good arguments”). That’s an important achievement in human phrases, in medical phrases, in scientific phrases, in each manner. We’ll see whether or not that’s the Twitter Musk preserves (as I do know the censorship-mad PMC won’t). We can even see how Twitter holds up below the World Cup stress check. Attention-grabbing occasions!


[1] These are the microservices Musk calls “bloatware.” Individually, maybe. Within the combination, I’m not so certain. For instance, if Twitter’s copyright strike system was a microservice, there’s a problem:

Social media customers on Twitter can now add total films in threads as the corporate’s automated copyright enforcement/ takedown system now not appears to be practical, Forbes reported on Sunday.

A person uploaded the 2006 film, The Quick and the Livid: Tokyo Drift in two-minute video clippings unfold out in a thread of 49 tweets, which when viral on the platform. It wasn’t a one-off occasion that in some way simply received via. One other person uploaded the film Hackers in a equally lengthy thread on Saturday, which even received 14,000 likes, Enterprise Insider reported.

Hollywood received’t like that. Nor will FIFA. However I’ve a suggestion:

I guess the pirated films wouldn’t go viral if the algo didn’t push them. Musk ought to flip off the algo! Give folks what they need: A timeline of tweets from accounts that they observe, in reverse chronological order. This could take away an infinite quantity of bloat.

[2] What I name a “yarn diagram” in conspiracy idea is a graph. The nodes are conspirators; the yarn is relationships. Once I say that the yarn in a yarn diagram is “too tight,” I imply that the creator of the graph hasn’t been nuanced sufficient about relationships (for instance, assuming orders are given when shared curiosity is sufficient).

[3] Or possibly it does. From former staffer Dan Luu:

One more reason to have in-house experience in numerous areas is that they simply pay for themselves, which is a particular case of the generic argument that giant firms needs to be bigger than most individuals anticipate as a result of tiny proportion good points are value a big quantity in absolute {dollars}. If, within the lifetime of the specialist staff just like the kernel staff, a single individual discovered one thing that persistently lowered TCO [Total Cost of Ownership] by 0.5%, that will pay for the staff in perpetuity, and Twitter’s kernel staff has discovered many such modifications. Along with kernel patches that typically have that form of influence, folks can even discover configuration points, and many others., which have that form of influence.

[4] If Musk is correct about headcount, and also you play the ponies:


Various Twitter customers have ventured out into Mastodon, Twitter’s open-source competitor. Naturally, they reported their findings on Twitter.

Mastodon is giant by its personal requirements, however not by Twitter’s:

There may be one Twitter; there are lots of Mastodon servers (therefore jokes about garages). Meaning onboarding is difficult. It’s a must to choose a server, then apply for an account, then wait. Some servers have shut down functions as a result of they will’t deal with the load;

The Mastodon servers are collectively referred to as the “fediverse” (federated universe):

The fediverse isn’t easy:

Typically members of the fediverse struggle:

After which there are the admins:

I’ve arrange an account on Mastodon. Whether or not I transfer my tiny presence there’s an open query…

Print Friendly, PDF & Email



Please enter your comment!
Please enter your name here

Most Popular

Recent Comments