The Netflix/Qwikster disaster

As usual, this post is merely an edition of comments I’ve made on HN.

The Lowdown

Netflix has announced its intention to spin out its DVD-by-mail service into a separate entity called “Qwikster”. Netflix will henceforth offer streaming video exclusively. The two sites are completely discrete and will no longer share data or even a billing mechanism.

This is a terrific disaster for Netflix devotees and Netflix itself.

The Customers’ Disaster

The primary issue is that Netflix has made a major consumer-facing split on what is really an implementation detail. Netflix users want to watch movies. That is the reason you get a Netflix account, that is the goal of the Netflix customer. Whether that movie is available on DVD or via the intertubes’ streaming fairies is not really exceptionally relevant to the customers’ ultimate goal of watching that movie. Netflix is a company for intrepid movie-watchers, and artificially restricting this to intrepid streamers is just leaving money on the table.

Netflix’s success heretofore has been based upon a vast simplification in watching movies. In splitting the service and creating an artificial rift in their offerings, they’ve backpedaled tremendously; with the hard division of DVD and streaming, Netflix has gone to lengths to de-simplify your movie watching in order to remedy what really was a problem with management structure.

Netflix emphasizes, as they have for years, that DVD-by-mail will eventually go away. They do this in the same breath as they attempt to encourage users to register for Qwikster, essentially promising that those who are still interested in receiving physical DVDs will, in the relatively near-term future, have even greater hassle to their ultimate goal of watching movies, because Qwikster will shut down and take its users’ ratings, recommendations, and rental history with it.

What Should Have Happened: Redbox + Netflix

Instead of dividing the company in a way that made movie-watching less convenient, Netflix should have turned its attention to Redbox. Netflix and Redbox (which is now owned by Coinstar, a much less compatible couple) are a match made in heaven; Redbox is ubiquitous in most areas of the US these days and could function well as Netflix’s physical distribution arm, cutting Netflix’s dependence on the dwindling US Postal Service and saving postal costs.

I am personally acquainted with several individuals whose Redbox usage has replaced Netflix. These people are primarily interested in recent-ish releases and might have streamed if the content were available for streaming, but since desired content is rarely streamable, found it simpler just to go to Redbox and pick up the physical DVD. This is much faster than waiting for the mail, which in most cases has a 2-4 day turnaround.

Redbox would be an investment worth quite a significant chunk of money and integrating a user’s Redbox experience with the Netflix website would have been a great win. My speculation is that Coinstar doesn’t really know what to do with Redbox and they may have acquired the kiosks in anticipation of reselling in the first place, since it really doesn’t fit in with their standard business practice.

I recognize that Reed Hastings would probably balk at this suggestion; Netflix doesn’t want any of the hassles of pesky physical media anymore, and if nothing else they’ve made that quite obvious today. But I think it’s misguided — as exciting as streaming is, I don’t think discs are going to make a permanent exit any time soon. There are still issues with streaming delivery, like conflicted, guarded ISPs (Comcast, whose cable subscription tallies are constantly diminished by Netflix) and the non-tech-savvy who insist on using ancient computers riddled with spyware and can barely get Gmail to load in less than 10 minutes, let alone stream a HD movie. It’s much easier to go pick up a disc and place it in a tray than it is to start a Netflix Instant movie, especially since one must usually install Silverlight before watching.

The staying power of physical DVDs is really its own post, so I’ll just stop there.

And on top of all that, the prevalent red color schemes of the Netflix website and the Redbox kiosk already match. How is this not obvious?

Reddit the open-source software

ketralnis has responded to this post here and throughout this thread.

Occurrences of “reddit the open-source software” have been abbreviated to “reddit OSS”. – Nov. 19, 2010

I use reddit, as in reddit the open-source software, for a website that doesn’t get much traffic for several reasons. reddit OSS is one of the bigger reasons. I want to talk about reddit OSS and its management for a moment.

reddit OSS is published at http://github.com/reddit and http://code.reddit.com. reddit.com usually works pretty well, but reddit OSS is very unfriendly to anyone that is not reddit.com.

There has not been a push for about a month, and before that, there had not been a push since mid-July, despite “planning on a much more sane release schedule for future patches (much closer to ‘weekly’ rather than ‘epoch modulo 10Ms’).” The long lag time between pushing changes makes code merges when a new version eventually does get pushed a serious undertaking, especially for those who run hobby or part-time sites (as most running the reddit open-source platform would be). Each time I have updated my reddit installation to a new HEAD it has been two or three days of configuration, re-merging, and bug-squashing before the updated codebase was working as expected; recently, subtle failures occurred while running ads for the site and essentially made it impossible to post comments. If changes were pushed in smaller increments, the same necessary merges would be much easier to handle; merging three or four changes is much simpler than merging 60-70+.

Merges get even more complicated because to customize reddit even in the most basic ways, you’ll have to hack up several base code files that contain a lot of other stuff. When you clone reddit from git, the clone comes with the same ads that run on reddit, and the only way to remove them is to edit that file, a file that git tracks and a file that clashes on merges (if you don’t –assume-unchanged, which is probably safe in this case as that file hasn’t been updated in over two years, but still extra hassle and excludes all future changes from applying automatically — changes which may be important).

There are several other instances for things that really should have been cleaned up for reddit OSS but still linger, and as you go through removing all them, you get quite a few changes built up — changes that cause problems when it’s time to pull. You shouldn’t have to sanitize the codebase of the OSS version in the first place; that’s the maintainer’s job.

Most obvious among these things that should have been stripped is the reddit alien. It is all over the place — under the submit link button, under the create a subreddit button, thumbnail placeholder, and so on. As far as I know the reddit alien is still held by Conde Nast/reddit corporate under an All Rights Reserved copyright license, as one might expect for a company’s logo. The term “reddit”, “subreddit”, etc., appear throughout the site, causing potential trademark liabilities.

If a website that runs reddit OSS starts to gain momentum, how long do we expect the lawyers at Conde Nast to abide usage of the reddit name and logo on a website over which they have no control, especially if that site infringes on reddit.com’s primary audience? Why can’t they draw an distinct alien for reddit OSS or just include generic images and icons from Tango et al? It would be a much better thing to do so. My site has been going for almost a year and I’m still finding the term “reddit” sprinkled in odd places, despite going through the translation file a few times. It’s hard-coded in some spots.

Then, to run reddit OSS, one must use memcached, Cassandra, an AMQP server like rabbitmq, PostgreSQL, and a handful of paster daemons included with reddit, which are currently configured to run with daemontools, so unless you want to spend a while converting the current scripts/daemons, you must also install and use daemontools. Furthermore, running these daemons is non-obvious and it was not required when I originally pulled, so it took me a while to figure out a lot of the weird bugs I got resulted from not running these daemons. These daemons are mostly for caching as far as I know, but if you don’t have them in place weird things like disappearing thumbnails and comments will befall you. The commit messages I saw did not make big shiny letters about it, and the overall documentation is poor.

reddit.com does almost no testing of reddit OSS. They just push out what they run on reddit.com. Many times in #reddit-dev I have seen “we haven’t tested it that way but it should work…” before someone describes a bug or submits a patch. reddit does not test reddit in a conventional environment.

In the October update, reddit merged several contributed patches, but prior thereto it was rather rare, only occurring a couple of times on a couple of patches (from the github history). There are still a lot of changes out there that would do well to be merged, but reddit.com is trying to keep the codebase unified (despite its super-ugly squash commits that get pushed out in the “weekly” updates), so if your patch would help most users of reddit OSS but not reddit.com, it won’t get merged. This can be good in some cases — it forced me to produce a more scalable database reconnect priority patch, for instance — but it can also mean that more sensible defaults or caching mechanisms for sites that are not reddit.com would be rejected.

The reddit guys insist that their number one priority is reddit.com and almost any time someone brings up a push of reddit.com to the OSS version or merging of a patch or whatever in #reddit-dev, ketralnis is adamant that there is just no time for that. reddit is clearly understaffed and reddit OSS is largely neglected.

There’s not necessarily anything wrong with that, but all of this means that reddit OSS is in prime condition for a fork. However, ketralnis does not think a fork is a good idea. Here is a snippet from IRC, with pieces omitted for brevity and coherence:

(01:03:12 AM) sjuxax: I am planning on forking reddit sometime soon fyi
(01:05:07 AM) ketralnis: I wouldn’t recommend that

(01:05:12 AM) sjuxax: why?

(01:05:21 AM) ketralnis: It’d be a nightmare to maintain against our code-releases, for one

(01:05:47 AM) ketralnis: For another, the license make it difficult to divorce from our brand

(01:05:55 AM) sjuxax: Well it’s already a nightmare to merge with the six-month release cycles and big changes you guys make.

(01:06:12 AM) ketralnis: Agreed, and we should do less of that

(01:06:14 AM) sjuxax: The license basically just requires the attribution at the bottom, right?

(01:06:39 AM) ketralnis: If you’re planning on forking it, you should actually read it. It’s not a long one

(01:06:43 AM) sjuxax: so we can leave that, but the alien is all over. Obviously the license won’t let us get rid of the powered by reddit logo, but the rest should be free to go

(01:06:51 AM) sjuxax: I have read it in the past, but it’s been a while

(01:07:16 AM) ketralnis: I understand where you’re coming from, but it would harm our open-source development to have it forked

(01:08:43 AM) sjuxax: Well I would prefer to keep upstream and the fork at least somewhat compatible

(01:08:58 AM) sjuxax: so hopefully most patches could still go both ways

(01:10:17 AM) sjuxax: but yeah, uh, sorry. reddit has neglected its open-source users imo so a fork is inevitable when you get serious users; that’s why we use OSS software; if the maintainer isn’t taking care of it, someone else can

(01:10:47 AM) ketralnis: We are taking care of it, in that it’s what’s running our live site, right now. 14 million pageviews yesterday.

(01:11:04 AM) ketralnis: So I’d say it’s holding up rather well under its current maintanence

(01:11:21 AM) sjuxax: OK, you are taking care of your reddit installation, you are running reddit for reddit which is fine if that’s what you want to do

(01:11:25 AM) ketralnis: The right solution is for me to set aside a day to merge up with public, not to go forking it

(01:11:28 AM) sjuxax: but it is not attractive as an option for not-reddit

(01:11:43 AM) ketralnis: I’m telling you, forking us will hurt reddit.

(01:11:50 AM) sjuxax: but you don’t set aside that day often enough; you were going to do it weekly but now it’s been months again

(01:12:57 AM) sjuxax: reddit as an open-source project is either going to get forked or going to continue to limp on. it will be nice for reddit’s reddit, but if things keep going how they have been going, virtually no one is going to use the code you publish.

(01:12:57 AM) ketralnis: I don’t have time to argue this right now. But trust me, you forking reddit will fuck up my week, and probably stall any future open source contribution to reddit.

(01:13:53 AM) ketralnis: Forking it will make that situation worse by losing the only developers *paid* to contribute to it from your fork, and any open source developers from either

So reddit corporate would not be happy to see a fork rise up, but what choice do users of reddit OSS have? Things are definitely not good the way they are now and I think that a fork is ultimately inevitable unless reddit revises their policies, allows some divergence, and finally takes the open-source side of things seriously.

Is there much interest in a fork out there? There’s lots of good contributions on github that remain unmerged, and a fork would be more active about merging these and especially merging changes that enhance the platform for smaller sites. Once someone gets reddit.com-level traffic, they can switch the platform to the official reddit OSS and then all of the onerous/tricky/annoying/monstrous stuff that is employed by reddit to allow caching and survival under that kind of traffic will be beneficial.

The paths before reddit.com/reddit corporate are A) take reddit OSS seriously, get patching and merging fixed up and make it easier to push out changes, and then maintain the open-source version frequently and well, including possible divergences where it benefits the OSS user; B) stay the course until someone forks, and its unclear what the ultimate consequences of this would be. ketralnis seems to think it would mean a secession of commercially-funded development entirely; or C) stay the course until everyone gives up on reddit OSS and the project withers and dies. What’ll it be?

Joel’s “Duct-Tape Programmer” is the only programmer you should ever hire.

I’ve just finished reading Joel Spolsky’s “Duct Tape Programmers” and jwz’s response to it.

These posts strike a chord with me, especially that Spolsky spends the entirety of the article praising “The Duct-Tape Programmer” before he urges you not to try it.

I’ve worked on teams where every member urgently insisted on rigid, absolute application of certain tools, citing familiarity as the main benefit. “When all you have is a hammer, everything looks like a nail.”

These teams tend to be weeks behind schedule, blinded by various unique brands of naivety, and in general this type of management produces an unwieldy monolith of a codebase, slow and non-intuitive.

jwz, on the other hand, is a pragmatic generalist who doesn’t fear learning a new thing if it is the right tool for the job. While others who blindly insist on the same old are motivated by self-interest, fear, and good performance reviews from uninformed business people using worse-than-useless metrics, the “duct-tape programmer”, epitomized by jwz, is concerned with performance, utility, and maintainability of the product, which sadly, often comes at the expense of high corporate positions reserved only for elite sycophants, not practical, usable realists. jwz’s blogs testify to this clearly.

When Mr. Spolsky implores readers not to follow in jwz’s footsteps of judgment and application, he promotes a pervasive negativity and self-decay within a development team. Obsession with fads or hyperfocus on one area of good practice and other symptoms of programming elitism do no good to someone actually interested in developing or maintaining functional software. Often “non-duct-tape” programmers can prattle shallow quips right off to back up their incompetence, but the fact is that attempts to squeeze a round peg through a square hole are always inadequate, even if the big bad “learning curve” must make an appearance. Invest the time up front and do it in a way that loosely approximates an intelligent approach instead of allowing politics and insecurities to hinder development.

I can’t tell you how many hours I’ve seen wasted bastardizing functions deep within the netherparts of CakePHP (for instance) when a faster, better, and more suitable custom component could have been built much faster and maintained much easier. A programmer that does that is an example of a bad programmer because no matter how well he knows the target platform, he’s going to regularly misapply that knowledge, and his technical religion that CakePHP (for instance) is ultimate, and everything must be done the Cake way (which, ironically, is tarnished by forcing yourself to modify core libs because you can’t not use Cake for something), impedes the macro-level goal of “build something that works well, reasonably quickly”.

Spolsky’s description paints the duct-tape programmer as an ignoramus, a programmer who’d just as well stick to pre-1990 languages, methods, and conventions, but that’s clearly not the case with jwz, who was one of the first third-party developers for the Palm Pre. jwz, like all other good programmers, simply knows how and when to avoid cruft and how and when to leverage existing work. This sense of decent judgment is gained only through long experience and technical exploration, the kind a “non-duct-tape” programmer is too afraid to get.

So, when Joel says the “duct-tape programmer”, what he really means is the “pragmatic, profitable programmer, the only kind of programmer you should ever hire or use; a ‘good’ programmer”. These programmers do follow the changes in the field and they do play with new technologies and methods. They simply know when to experiment, when to complicate, when to simplify, and when to ship.

Do yourself a favor and pattern your next hire after jwz; get a programmer with the know-how and the wisdom, because one is useless without the other.