November 13, 2014

Robert O'Callahan -- Relax, Scaling User Interfaces By Non-Integer Scale Factors Is Okay

About seven years ago we implemented "full zoom" in Firefox 3. An old blog post gives some technical background to the architectural changes enabling that feature. When we first implemented it, I expected non-integer scale factors would never work as well as scaling by integer multiples. Apple apparently thinks the same, since (I have been told) on Mac and iOS, application rendering is always to a buffer whose size is an integer multiple of the "logical window size". GNOME developers apparently also agree since their org.gnome.desktop.interface scaling-factor setting only accepts integers. Note that here I'm conflating user-initiated "full zoom" with application-initiated "high-DPI rendering"; technically, they're the same problem.

Several years of experience shows I was wrong, at least for the Web. Non-integer scale factors work just as well as integer scale factors. For implementation reasons we restrict scale factors to 60/N for positive integers N, but in practice this gives you a good range of useful values. There are some subtleties to implementing scaling well, some of which are Web/CSS specific.

For example, normally we snap absolute scaled logical coordinates to screen pixels at rendering time, to ensure rounding error does not accumulate; if the distance between two logical points is N logical units, then the distance between the rendered points stays within one screen pixel of the ideal distance S*N (where S is the scale factor). The downside is that a visual distance of N logical units may be rendered in some places as ceil(S*N) screen pixels and elsewhere as floor(S*N) pixels. Such inconsistency usually doesn't matter much but for CSS borders (and usually not other CSS drawing!), such inconsistent widths are jarring. So for CSS borders (and only CSS borders!) we round each border width to screen pixels at layout time, ensuring borders with the same logical width always get the same screen pixel width.

I'm willing to declare victory in this area. Bug reports about unsatisfactory scaling of Web page layouts are very rare and aren't specific to non-integral scale factors. Image scaling remains hard; the performance vs quality tradeoffs are difficult --- but integral scale factors are no easier to handle than non-integral.

It may be that non-Web contexts are somehow more inimical to non-integral scaling, though I can't think why that would be.

Full Article

November 13, 2014 11:48 AM


I saw Interstellar today. I enjoyed it --- it was beautiful and thought-provoking --- but about an hour in, it just stops making sense. I don't just mean the physics is bad, though it is, but the narrative fails to hold together on its own terms. Even so I think it's worth seeing on the big screen.

Apparently Jonathan Nolan wants to make one or more movies based on the Foundation series. That doesn't seem like a good idea to me; Foundation is an excellent series (at least the original trilogy; Asimov's later additions are embarrassing) but it's big ideas connected by threadbare narrative and could only make the barest bones of a watchable movie.

I can think of many science fiction stories more amenable to movie treatment. I've recently re-read (or more accurately re-skimmed) Marooned In Realtime by Vernor Vinge, and I think it's actually better than his more famous Hugo winners. It's packed full of big ideas but is genuinely moving in a way I don't get from his other books. Better still, from a movie point of view, there's a coherent narrative in the framework of a police procedural that makes it an easy story to tell. Plus exotic creatures and locations, robots, and space combat. What more could you want?

Full Article

November 13, 2014 09:11 AM

November 03, 2014

Robert O'Callahan -- HTML5 Video Correctness Across Browsers

A paper presents some results of cross-browser testing of HTML5 video. This sort of work is very helpful since it draws attention away from browser vendor PR efforts and towards the sort of issues that really impact Web developers, with actionable data.

Having said that, it's also very interesting to compare how browsers fare on correctness benchmarks designed without a browser-vendor axe to grind. Gecko usually does very well on such benchmarks and this one is no exception. If you read through the paper, we get almost everything right except a few event-firing issues (that I don't understand yet), more so than the other browsers.

Of course no test suite is perfect, and this one misses the massive Chrome canplaythrough bug: Chrome doesn't really implement canplaythrough at all. It simply fires canplaythrough immediately after canplay in all circumstances. That's still causing problems for other browsers which implement canplaythrough properly, since some Web sites have been written to depend on canplaythrough always firing immediately.

Still, this is good work and I'd like to see more like it.

Full Article

November 03, 2014 11:20 PM

November 02, 2014

Robert O'Callahan -- Auckland Half Marathon --- Barefoot

I ran the Auckland half-marathon today, barefoot all the way. It went well; I slightly improved my time compared to last year when I wore Vibram foot-gloves. I didn't see any other barefoot runners last year or this year, and maybe there will never be any others, but here are some tips just in case:

  • Nothing in the rules says you have to wear shoes and I didn't have any trouble from any officials.
  • The race has detailed instructions for tying the timing chip into your shoelaces and dire warnings should those instructions not be followed. Before the race I experimented with a few different ways to attach the chip to my foot, but they all failed. Today I just put the chip in my pocket and at the chip-readers (at the start line, the top of the Auckland Harbour Bridge, and the finish line) I held the chip in my hand as low to the ground as I could as I ran over the chip-reading system. I wasn't sure if that would work, but it did.
  • The road surface was reasonably smooth most of the way along the route. The section from Smale's Farm along the motorway until getting off the Harbour Bridge --- several kilometres --- is rougher than the rest but tolerable. There are sections around Halsey Street near the end which are very rough, but they're quite short. Overall the surface was better than I usually get running on footpaths.
  • My feet hurt less at the end than they did when I wore the Vibrams last year. Last year my feet got really sweat-soaked so that may have had something to do with it.
  • I got a lot of good-natured comments from runners and bystanders. Some people probably thought I was a nutter, but to be fair, they're not clearly wrong.
  • Aid stations were surrounded by discarded cups of Powerade, making the road sticky with Powerade underfoot. Icky, but it quickly rubs off :-).

To the question of how you get into shape to run a half-marathon barefoot in the first place, I have little to say. I always run barefoot and I run home from work most days.

Overall it was a great experience and I expect I'll do it again next year.

Full Article

November 02, 2014 08:22 AM

October 27, 2014

Robert O'Callahan -- Are We Fast Yet? Yes We Are!

Spidermonkey has passed V8 on Octane performance on arewefastyet, and is now leading V8 and JSC on Octane, Sunspider and Kraken.

Does this matter? Yes and no. On one hand, it's just a few JS benchmarks, real-world performance is much more complicated, and it's entirely possible that V8 (or even Chakra) could take the lead again in the future. On the other hand, beating your competitors on their own benchmarks is much more impressive than beating your competitors on benchmarks which you co-designed along with your engine to win on, which is the story behind most JS benchmarking to date.

This puts us in a position of strength, so we can say "these benchmarks are not very interesting; let's talk about other benchmarks (e.g. asm.js-related) and language features" without being accused of being sore losers.

Congratulations to the Spidermonkey team; great job!

Full Article

October 27, 2014 10:58 PM

October 19, 2014

Robert O'Callahan -- Pinnacles Tramp #2

During the weekend my children and I, plus some people from church and a friend, went tramping to the Pinnacles hut in Kauaeranga Valley in the Coromandel peninsula (near Thames, about two hours drive from Auckland). The weather was wet but the misty environment had its own delicate beauty. We only got rained on for about 45 minutes of the walk so everyone still had a good time. We played some good games of Bang and Dutch Blitz and, since it was only a single night, we brought real food and ate well!

Full Article

October 19, 2014 09:24 PM

Photos From North America

Full Article

October 19, 2014 09:14 PM

October 14, 2014

Robert O'Callahan -- Back In New Zealand

I just finished a three-week stint in North America, mostly a family holiday but some work too. Some highlights:

  • Visited friends in Vancouver. Did the Grouse Grind in just over an hour. Lovely mountain.
  • Work week in Toronto. Felt productive. Ran barefoot from downtown to Humber River and back a couple of times. Lovely.
  • Rendezvoused with my family in New York. Spent a day in White Plains where we used to live, and at Trinity Presbyterian Church where we used to be members. Good sermon on the subject of "do not worry", and very interesting autobiographical talk by a Jewish Christian. Great time.
  • Visited the 9/11 Museum. Very good, though perhaps a shade overstressing the gravity of 3000 lives lost. One wonders what kind of memorial there will be if a nuke kills 100x that many.
  • Our favourite restaurant in Chinatown, Singapore Cafe, is gone :-(.
  • Had some great Persian food :-).
  • The amazingness of New York is still amazing.
  • Train to Boston. Gave a talk about rr at MIT, hosted by my former supervisor. Celebrated 20-year anniversary of me starting as his first (equal) grad student. Had my family watch Dad at work.
  • Spent time with wonderful friends.
  • Flew to Pittsburgh. More wonderful friends. Showed up at our old church with no prior warning to anyone. Enjoyed reactions. God continues to do great things there.
  • La Feria and Fuel-and-Fuddle still great. Still like Pittsburgh a lot.
  • Flew to San Francisco. Late arrival due to flight being rerouted through Dallas, but did not catch Ebola.
  • Saw numerous of seals and dolphins from the Golden Gate Bridge.
  • Showed my family a real Mozilla office.
  • Two days in Mountain View for Gecko planning meetings. Hilarious dinner incident. Failed to win at Settlers.
  • Took family to Big Basin Redwoods State Park; saw pelicans, deer, a dead snake, a banana slug, and a bobcat.
  • Ever since we made liquid nitrogen ice cream for my bachelor party, I've thought it would make a great franchise; Smitten delivers.
  • Kiwi friends in town for Salesforce conference; took them to Land's End for a walk. Saw a coyote.
  • Watched Fleet Week Blue Angels display from Twin Peaks. Excellent.
  • Played disc golf; absolutely hopeless.
  • Went to church at Home of Christ #5 with friends. Excellent sermon about the necessity of the cross.
  • Flew home on Air NZ's new 777. Upgraded entertainment system is great; more stuff than you could ever hope to watch.

Movie picoreviews:

    Edge Of Tomorrow: Groundhog Day meets Starship Troopers. Not as good as Groundhog Day but pretty good.

    X-Men: Days Of Future Past: OK.

    Godzilla: OK if your expectations are set appropriately.

    Dawn Of The Planet Of The Apes: watched without sound, which more or less worked. OK.

    Amazing Spider-Man 2: Bad.

    Se7en: Good.

Full Article

October 14, 2014 02:37 AM

September 29, 2014

Robert O'Callahan -- Upcoming rr Talk

Currently I'm in the middle of a 3-week visit to North America. Last week I was at a Mozilla graphics-team work week in Toronto. This week I'm mostly on vacation, but I'm scheduled to give a talk at MIT this Thursday about rr. This is a talk about the design of rr and how it compares to other approaches. I'll make the content of that talk available on the Web in some form as well. Next week I'm also mostly on vacation but will be in Mountain View for a couple of days for a planning meeting. Fun times!

Full Article

September 29, 2014 12:59 AM

September 08, 2014

Robert O'Callahan -- rr 2.0 Released

Thanks to the hard work of our contributors, rr 2.0 has been released. It has many improvements over our 1.0 release:

  • gdb's checkpoint, restart and delete checkpoint commands are supported.
    These are implemented using new infrastructure in rr 2.0 for fast cloning of replay sessions.
  • You can now run debuggee functions from gdb during replay.
    This is a big feature for rr, since normally a record-and-replay debugger will only replay what happened during recording --- and of course, function calls from gdb did not happen during recording. So under the hood, rr 2.0 introduces "diversion sessions", which run arbitrary code instead of following a replay. When you run a debuggee function from gdb, we clone the current replay session to a diversion session, run your requested function, then destroy the diversion and resume the replay.
  • Issues involving Haswell have been fixed. rr now runs reliably on Intel CPU families from Westmere to Haswell.
  • Support for running rr in a VM has been improved. Due to a VMWare bug, rr is not as reliable in VMWare guests as in other configurations, but in practice it still works well.
  • Trace compression has been implemented, with compression ratios of 5-40x depending on workload, dramatically reducing rr's storage and I/O usage.
  • Many many bugs have been fixed to improve reliability and enable rr to handle more diverse workloads.

All the features normally available from gdb now work with rr, making this an important milestone.

The ability to run debuggee functions makes it much easier to use rr to debug Firefox. For example you can dump DOM, frame and layer trees at any point during replay. You can debug Javascript to some extent by calling JS engine helpers such as DumpJSStack(). Some Mozilla developers have successfully used rr to fix real bugs. I use it for most of my Gecko debugging --- the first of my research projects that I've actually wanted to use :-).

Stephen Kitt has packaged rr for Debian.

Considerable progress has been made towards x86-64 support, but it's not ready yet. We expect x86-64 support to be the next milestone.

I recorded a screencast showing a quick demo of rr on Firefox:

Full Article

September 08, 2014 12:39 PM

VMWare CPUID Conditional Branch Performance Counter Bug

This post will be uninteresting to almost everyone. I'm putting it out as a matter of record; maybe someone will find it useful.

While getting rr working in VMWare guests, we ran into a tricky little bug. Typical usage of CPUID. e.g. to detect SSE2 support, looks like this pseudocode:

CPUID(0); // get maximum supported CPUID subfunction M
if (S <= M) {
CPUID(S); // execute subfunction S
Thus, CPUID calls often occur in pairs with a conditional branch between them. The bug is that in a VMWare guest, when we count the number of conditional branches executed, the conditional branch between those two CPUIDs is usually (but not always) omitted from the count. We assume this is a VMWare bug because it does not happen on the same hardware outside of a VM, and it does not happen in a KVM-based VM.

Experiments show that some code sequences trigger the bug and other equivalent sequences don't. Single-stepping and other kinds of interference suppress the bug. My best guess is that VMWare optimizes some forms of the above code, perhaps to reduce the number of VM exits, and in so doing skips execution of the conditional branch, without taking into account that this might perturb performance counter values. Admittedly, it's unusual for software to rely on precise performance counter values the way rr does.

This sucks for rr because rr relies on these counts being accurate. We sometimes find that replay diverges because one of these conditional branches was not counted during recording but is counted during replay. (The other way around is possible too, but less frequently observed.) We have some heuristics and workarounds, but it's difficult to fully work around without adding significant complexity and/or slowdown.

The bug is easily reproduced: just use rr to record and replay anything simple. When replaying, rr automatically detects the presence of the bug and prints a warning on the console:

rr: Warning: You appear to be running in a VMWare guest with a bug
where a conditional branch instruction between two CPUID instructions
sometimes fails to be counted by the conditional branch performance
counter. Partial workarounds have been enabled but replay may diverge.
Consider running rr not in a VMWare guest.

Steps forward:

  • Find a way to report this bug to VMWare.
  • Linux hosts can run rr in KVM-based VMs or directly on the host. Xen VMs might work too.
  • Parallels apparently supports PMU virtualization now; if Parallels doesn't have this bug, it might be the best way to run rr on a Mac or Windows host.
  • We can add a "careful mode" that would probably almost always replay successfully, albeit with additional overhead.
  • The bug is less likely to show up once rr supports x86-64. At least in Firefox, CPUID instructions are most commonly used to detect the presence of SSE2, which is unnecessary on x86-64.
  • In practice, recording Firefox in VMWare generally works well without hitting this bug, so maybe we don't need to invest a lot in fixing it.

Full Article

September 08, 2014 11:37 AM

August 11, 2014

Robert O'Callahan -- Milestones On The Road To Christianity

Around the age of 20 I found myself struggling with some fairly deep philosophical questions. The most important was this: assuming (as I did) naturalism is true, then what should I do?

It seemed clear to me then (and still does) that if naturalism is true, the is-ought problem is insurmountable. There can be no objective moral truths or goals. The best we can do is identify commonly held moral values and pursue them. Unfortunately --- if honesty is one of those values --- we cannot tell others that their behavior is, in any objective sense, wrong. For example, we observe that Hitler's moral opinions are different from ours, but we could not claim that our moral opinions are intrinsically more valid. All we could do is wage war against him and hope our side prevails. Might makes right.

That doesn't make naturalism incoherent, but it opens a chasm between what naturalists can really believe about moral statements and the way almost everyone uses them in practice. The more die-hard naturalists are prone to say things like "naturalism is true, and therefore everyone should ... (stop believing in God, etc)" without respecting the limitation that the consequent ought-statements are subjective opinions, not objectively rational facts. It's really very difficult to be a proper moral relativist through-and-through!

Making this all much more difficult was my awareness of being able to reshape my own moral opinions. The evolutionary-psychology approach of "these are the values imbued by my primate brain; work them out" seems totally inadequate when the rational part of my brain can give priority to any subset of values (or none) and use that as justification for rewriting the others. Given a real choice between being a hero and a monster, on what grounds can one make that decision? It seemed a bit narrow-minded to reject monstrosity simply because it was less popular.

This all made me very dissatisfied with naturalism as a worldview. If it's true, but is powerless to say how one should live --- indeed, denies that there can be any definitive guidance how to live --- it's inadequate. Like a scientific theory that lacks predictive power, whether it's true or not, one has to keep looking for more.

(OK, I was a weird kid, but everyone thinks about this, right?)

Full Article

August 11, 2014 10:16 AM

August 08, 2014

Robert O'Callahan -- cf1e5386ecde9c2eb9416c9b07416686

Full Article

August 08, 2014 05:39 AM

Choose Firefox Now, Or Later You Won't Get A Choice

I know it's not the greatest marketing pitch, but it's the truth.

Google is bent on establishing platform domination unlike anything we've ever seen, even from late-1990s Microsoft. Google controls Android, which is winning; Chrome, which is winning; and key Web properties in Search, Youtube, Gmail and Docs, which are all winning. The potential for lock-in is vast and they're already exploiting it, for example by restricting certain Google Docs features (e.g. offline support) to Chrome users, and by writing contracts with Android OEMs forcing them to make Chrome the default browser. Other bad things are happening that I can't even talk about. Individual people and groups want to do the right thing but the corporation routes around them. (E.g. PNaCl and Chromecast avoided Blink's Web standards commitments by declaring themselves not part of Blink.) If Google achieves a state where the Internet is really only accessible through Chrome (or Android apps), that situation will be very difficult to escape from, and it will give Google more power than any company has ever had.

Microsoft and Apple will try to stop Google but even if they were to succeed, their goal is only to replace one victor with another.

So if you want an Internet --- which means, in many ways, a world --- that isn't controlled by Google, you must stop using Chrome now and encourage others to do the same. If you don't, and Google wins, then in years to come you'll wish you had a choice and have only yourself to blame for spurning it now.

Of course, Firefox is the best alternative :-). We have a good browser, and lots of dedicated and brilliant people improving it. Unlike Apple and Microsoft, Mozilla is totally committed to the standards-based Web platform as a long-term strategy against lock-in. And one thing I can say for certain is that of all the contenders, Mozilla is least likely to establish world domination :-).

Full Article

August 08, 2014 01:33 AM

July 16, 2014

Robert O'Callahan -- Multiverses And Anthropic Reasoning

I liked this article summarizing the current state of science regarding multiverse theories. It's very clear and well-illustrated, and, as far as I know, accurate.

This quote is particularly interesting:

So as appealing as the idea is that there are other Level 1 Multiverses out there with different constants than our own, we have good physical reasons based on observable evidence to think it’s unlikely, and zero good reasons (because wanting it to be so is not a good reason) to think it’s likely.

He doesn't mention why anyone would "want it to be so", i.e. believe that other universes of a "Level 1 Multiverse" could have different constants to our own. However, I'm pretty sure he had in mind the selection-bias explanation for the anthropic coincidences. That is, if we accept that only a narrow range of possible values for the fundamental physical constants are compatible with the existence of intelligent life (and most scientists do, I think), then we would like to be able to explain why our universe's constants are in that range. If there are an abundance of different universes, each with different values for the physical constants, then most of them would be dead but a lucky few would sustain intelligent life, and naturally we can only observe one of those latter.

This reasoning relies on the assumption that there are abundance of different universes with different values for the physical constants. Scientists obviously would prefer to be able to deduce this from observations rather than pull it out of thin air. As discussed in the above article, theories of chaotic inflation --- which are reasonably well-grounded in observations of our own universe --- predict the existence of alternate universes. If those universes could have different values for physical constants (or even different physical laws), we'd have an observationally-grounded theory that predicts exactly the kind of multiverse needed to power the selection-bias explanation for the anthropic coincidences. Unfortunately for proponents of that explanation, the science isn't working out.

Of course, the selection-bias explanation could still be valid, either because new information shows that chaotic-inflation universes can get different constants after all, or because we assume another level of multiverse, whose existence is not due to chaotic inflation. However, many scientists (such as in the article above) find the assumption of a higher-level multiverse quite unsatisfactory.

Unsurprisingly, I'm comfortable with the explanation that our universe was intentionally created for intelligent life to live in. Incidentally, you don't need to be a classical theist to adopt this explanation; some atheist philosophers argue with varying degrees of seriousness that we are (probably) living in a simulation.

Full Article

July 16, 2014 01:29 PM

July 02, 2014

Robert O'Callahan -- Implementing Scroll Animations Using Web Animations

It's fashionable for apps to perform fancy animations during scrolling. Some examples:

  • Parallax scrolling
  • Sticky headers that scroll normally until they would scroll out of view and then stop moving
  • Panels that slide laterally as you scroll vertically
  • Elements that shrink as their available space decreases instead of scrolling out of view
  • Scrollable panels that resist scrolling as you get near the end

Obviously we need to support these behaviors well on the Web. Also obviously, we don't want to create a CSS property for each of them. Normally we'd handle this diversity by exposing a DOM API which lets developers implement their desired behavior in arbitrary Javascript. That's tricky in this case because script normally runs on the HTML5 event loop which is shared with a lot of other page activities, but for smooth touch tracking these scrolling animation calculations need to be performed reliably at the screen refresh rate, typically 60Hz. Even for skilled developers, it's easy to have a bug where once in a while some page activity (e.g. an event handler working through some unexpected large data set) blows the 16ms budget to make touch dragging less than perfect, especially on low-end mobile devices.

There are a few possible approaches to fixing this. One is to not provide any new API, hope that skilled developers can avoid blowing the latency budget, and carefully engineer the browser to minimize its overhead. We took this approach to implementing homescreen panning in FirefoxOS. This approach sounds fragile to me. We could make it less fragile by changing event dispatch policy during a touch-drag, e.g. to suppress the firing of "non-essential" event handlers such as setTimeouts, but that would add platform complexity and possibly create compatibility issues.

Another approach would be to move scroll animation calculations to a Worker script, per an old high-level proposal from Google (which AFAIK they are not currently pursuing). This would be more robust than main-thread calculations. It would probably be a bit clumsy.

Another suggestion is to leverage the Web's existing and proposed animation support. Basically we would allow an animation on an element to be use another element's scroll position instead of time as the input to the animation function. Tab Atkins proposed this with declarative CSS syntax a while ago, though it now seems to make more sense as part of Web Animations. This approach is appealing because this animation data can be processed off the main thread, so these animations can happen at 60Hz regardless of what the main thread is doing. It's also very flexible; versions of all of the above examples can be implemented using it.

One important question is how much of the problem space is covered by the Web Animations approach. There are two sub-issues:

  • What scrolling effects depend on more than just the scroll position, e.g. scroll velocity? (There certainly are some, such as headers that appear when you scroll down but disappear when you scroll up.)
  • For effects that depend on just the scroll position, which ones can't be expressed just by animating CSS transforms and/or opacity as a function of the scroll position?
If people are aware of scrolling effects in either of those two categories, it would be very useful to hear about them.

Full Article

July 02, 2014 09:10 PM

May 25, 2014

Robert O'Callahan -- Unnecessary Dichotomy

As the scope of the open Web has expanded we've run into many hard issues such as DRM, support for patented video codecs, and applications needing APIs that increase fingerprintability. These issues are easily but incorrectly framed as choices between "principles" and "pragmatism" --- the former prioritizing Mozilla's mission, the latter prioritizing other things such as developer friendliness for the Web platform and Firefox market share. This framing is incorrect because the latter are essential components of our strategy for pursuing our mission.

For example I believe the optimal way to pursue our goal of unencumbered video codecs is neither to completely rely on platform video codecs (achieving nothing for our goal) nor to refuse to support all patent-encumbered codecs (in the current market, pushing almost all users and developers to avoid Firefox for video playback). Instead it is somewhere in between --- hopefully somewhere close to our current strategy of supporting H.264 in various ways while we support VP8/VP9 and develop an even better free codec, Daala. If we deliberately took a stance on video that made us irrelevant, then we would fail to make progress towards our goal and we would have actually betrayed our mission rather than served it.

Therefore I do not feel any need to apologize for our positions on encumbered codecs, DRM and the like. The positions we have taken are our best estimate of the optimal strategy for pursuing our mission. Our strategy will probably turn out to be suboptimal in some way, because this is a very difficult optimization problem, requiring knowledge of the future ... but we don't need to apologize for being unomniscient either.

A related problem is that our detractors tend to view our stance on any given issue in isolation, whereas we really face a global optimization problem that spans a huge range of issues. For example, when developers turn away from the Web platform for any reason, the Web as a whole is diminished, and likewise when users turn away from Firefox for any reason our influence of most issues is diminished. So the negative impact of taking a "hyper-principled" stand on, say, patent-encumbered video codecs would be felt across many other issues Mozilla is working on.

Having said all that, we need to remain vigilant against corruption and lazy thinking, so we need keep our minds open to people who complain we're veering too much in one direction or another. In particular I hope we continue to recruit contributors into our community who disagree with some of the things we do, because I find it much easier to give credence to contributors than to bystanders.

Full Article

May 25, 2014 11:29 PM

May 23, 2014

Robert O'Callahan -- Against The "Internet Of Things"

I was lucky to be at the Berkeley "programming systems" retreat in Santa Cruz a couple of weeks ago. One topic that came up was programming in the world of the "Internet of Things" --- the anticipated future where everyone has dozens of very small, very low-power devices with CPUs, sensors and radios. There's certainly a lot of interesting work to be done in that area, but nobody seems to have asked the question "do we really want an Internet of Things?" Because I don't, not until we've addressed the pervasive problems we already have with security, privacy, and general untrustworthiness of our computing infrastructure. It's obvious that these buy-and-forget devices won't get timely security updates (if updates are even adequate for security), so governments and criminal organizations will be battling for control over them, out of sight and mind of their nominal owners. So we're talking about making the surveillance network of the NSA (and even worse actors) a lot more pervasive. These devices will be able to run for years without a battery change, so when you buy a new house or car, you will inherit a miasma of unseen machines doing everyone's bidding but yours.

There's going to be a big market for scanning and cleaning tools. A small handheld EMP unit would be great. Someone should get on this problem.

Full Article

May 23, 2014 05:04 AM

May 04, 2014

Robert O'Callahan -- Milford Track

I took last week off to walk the Milford Track with some family members (plus a few days in Queenstown). The Milford Track is the most famous multi-day "Great Walk" in New Zealand and you have to book months in advance to do it during the regular season. It deserves every bit of its hyping as the finest walk in the world.

This three-night, four-day walk can be done "guided" or "independently". You pay a lot of money for the "guided" version and in return you stay in lodges with clean sheets, hot showers, and catered meals. For a lot less money the "independent" version has you staying at fairly standard (but slightly fancier than average) Department of Conservation huts, which means no showers and you carry your own food and bedding. We did the independent version. With both versions you must walk in the westward direction and stop for three nights in the designated hut/lodge each night. Each DOC hut has a resident ranger, and they're all brilliant to talk to.

You start by taking a boat to the north end of Lake Te Anau, then walking for just over an hour to the first hut. It's such a short distance you can carry whatever luxuries you can consume on that first night --- sausages, bacon and eggs were popular items amongst our fellow trampers. Then you work your way up to the end of the Clinton Valley, cross Mckinnon pass on the third day, and walk down Arthur Valley to Milford Sound to be transported by boat to the village of Milford itself.

The scenery is, needless to say, astonishing --- mountains, lakes, trees, rivers, waterfalls, and all that. The environment is pristine; every stream and river is crystal clear where not colored by beech tannins, and all are perfectly drinkable. There's fascinating wildlife --- trout, keas, wekas, black robins. The weather is highly variable --- sunshine, rain and snow are all quite common and we had them all in our four days (very little snow though). Having rain near the beginning of the walk, like we did, is excellent because it powers up all the streamlets and waterfalls.

One curious fact is that the Mckinnon Pass area was first explored by Europeans specifically to find a route for tourists to cross overland to Milford. Almost immediately after the pass was first crossed, tourists started using the Milford Track, often in rather appalling conditions by today's standards. For many years there was no road out of Milford so track walkers arriving in Milford generally had to walk straight back out again.

Full Article

May 04, 2014 11:58 AM

April 09, 2014

Robert O'Callahan -- Getting Back To Work

... is what we need now. So let me give a brief summary of what's happening with me work-wise.

Last year I fully divested my direct reports. No more management work to do, yay! I think they probably all ended up with a better manager than they had.

I've been involved in a lot of different projects and a lot of helping other people get their work done. In between I managed to carve out a few things of my own:

  • I built reftests for async panning to try to stem the tide of regressions we were encountering there. Unfortunately that's not quite done because most of those tests aren't running on TBPL yet.
  • I worked on CSS scroll snapping with an intern. Unfortunately the spec situation got bogged down; there's an impasse between us and Microsoft over the design of the CSS feature, and Google has decided not to do a CSS feature at all for now and try to do it with JS instead. I'm skeptical that will work will, and looking forward to their proposal, but it's taking a while.
  • I landed an implementation of the CSS OM GeometryUtils interface, described in blog posts here and here. This fixes a functionality gap in the Web platform and was needed by our devtools team. Almost everything you would need to know about the CSS box geometry of your page is now easy to get.
  • Lately I've been doing work on rr. People are trying to use it, and they've been uncovering bugs and inconveniences that Chris Jones and I have been fixing as fast as we can. Terrence Cole used it successfully to help fix a JS engine bug! I'm using it a lot myself for general-purpose debugging, and enjoying it. I want to spend a few more days getting some of the obvious rough edges we've found filed off and then make a new release.

Looking forward, I expect to be working on making our async scrolling fully generic (right now there are some edge cases where we can't do it), and working on some improvements to our MediaStreamGraph code for lower latency video and audio.

Full Article

April 09, 2014 11:54 AM

Fighting Media Narratives

... is perhaps futile. A lot of what I have to say has already been said. Yet, in case it makes a difference:

  • Almost all Mozilla staff supported keeping Brendan Eich as CEO, including many prominent LGBT staff, and many made public statements to that effect. A small number of Tweeters calling for him to step down got all the media attention. The narrative that Mozilla staff as a group "turned against Brendan" is false. It should go without saying, but most Mozilla staff, especially me, are very upset that he's gone. I've known him, worked with him and fought alongside him (and sometimes against him :-) ) for fourteen years and having him ripped away like this is agonizing.
  • The external pressure for Brendan to step down was the primary factor driving the entire situation. The same issue exploded in 2012 but there was less pressure and he got through it. No doubt Mozilla could have handled it better but the narrative that blames Mozilla for Brendan's departure misses the big picture. Boycotting Mozilla (or anyone for that matter) for cracking under intense pressure is like shooting a shell-shocked soldier.
  • As a Christian, Mozilla is as friendly a workplace as any tech organization I've known --- which is to say, not super friendly, but unproblematic. Because of our geographic spread --- not just of headcount, but of actual power --- and our broad volunteer base I think we have more real diversity than many of our competitors. The narrative that Mozilla as a group has landed on one side of the culture war is false, or at least no more true than for other tech organizations. In fact one thing I've really enjoyed over the last couple of weeks is seeing a diverse set of Mozilla people pull together in adversity and form even closer bonds.

I'll also echo something else a lot of people are saying: we have to fix Internet discourse somehow. It's toxic. I wrote about this a while back, and this episode has made me experience the problem at a whole new level. I'll throw one idea out there: let's communicate using only recorded voice+video messages, no tweets, no text. If you want to listen to what I have to say, you have to watch me say it, and hopefully that will trigger some flickers of empathy. If you want me to listen to you, you have to show me your face. Want to be anonymous, do it the old-fashioned way and wear a mask. Yeah I know we'd have to figure out searchability, skimmability, editing, etc etc. Someone get to work on it.

Full Article

April 09, 2014 11:31 AM

Mozilla Matters

How much does the world need Mozilla? A useful, if uncomfortable, thought experiment is to consider what the world would be like without Mozilla.

Consider the world of Web standards. Microsoft doesn't contribute much to developing new Web features, and neither does Apple these days. Mozilla and Google do. Google, per Blink's own policy (mirroring our own), relies on feedback and implementation by other browser vendors, i.e. usually us. If you take Mozilla out of the equation, it's going to be awfully hard to apply the "two independent implementations" test for new Web features. (Especially since Webkit and Blink still have so much shared heritage.) For example it's hard to see how important stuff like Web Audio, WebGL and WebRTC would have become true multi-vendor standards without us. Without us, most progress would depend on unilateral extensions by individual vendors. That has all the downsides of a single-implementation ecosystem --- a single implementation's bugs become the de-facto standard; the standards process, if there even is one, becomes irrelevant; and even more power accrues to the dominant vendor.

In the bigger picture, it would be dangerous to leave the Web --- and the entire application platform space --- in the hands of three very large US corporations who have few scruples and who each have substantial non-Web interests to protect, including proprietary desktop and mobile platforms. It has always been very important that there be a compelling vendor-neutral platform for people to deploy content and apps on, a platform without gatekeepers and without taxes. The Web is that platform ---- for now. Mozilla is dedicated to preserving and enhancing that, but the other vendors are not.

Mozilla has plenty of faults, and lots of challenges, but our mission is as important as ever ... probably more important, given how computing devices and the Internet penetrate more and more of our lives. More than ever, pursuing our mission is the greatest good I can do with the talents God has given me.

Full Article

April 09, 2014 10:34 AM

April 06, 2014

Robert O'Callahan -- Responsible Self-Censorship

People may be wondering why I, as one of the most notorious Christians at Mozilla, have been silent during the turmoil of recent days. It is simply because I haven't been able to think of anything to say that won't cause more harm in the current environment. It is not because I am afraid, malicious, or suffering from a lack of freedom of speech in any way. As soon as I have the environment and the words to say something helpful, I will.

By "the current environment" I mean the situation where many of the culture warriors of all stripes are picking over every utterance looking for something to be angry against or something to fuel their anger against others.

Update Actually, right after posting that, I thought of something to say.

I have never loved the people of Mozilla as much as I do right now. With just a few exceptions, your commitment and grace under fire is inspiring and (in the old-fashioned sense) awesome. I'm glad I'm here for this.

Full Article

April 06, 2014 09:27 PM

March 26, 2014

Robert O'Callahan -- Conflict

I was standing in a long line of people waiting to get their passports checked before departing Auckland to the USA. A man pushed into the line ahead of me. People around me sighed and muttered, but no-one did anything. This triggered my "if no-one else is willing to act, it's up to me" instinct, so I started a conversation:

Roc: Shouldn't you go to the back of the line?
Man: I'm in a hurry to get to my flight.
There are only two flights to the USA in the immediate future and AFAIK neither of them is imminent.
Roc: Which flight are you on?
Man: What's the big deal? I'm hardly going to slow you down.
Roc: But it's rude.
Man: So what do you want me to do?
Roc: I want you to move to the back of the line.
Man: You're going to have to make me.

At that point I couldn't think of anything to say or do so I let it go. It turned out that he was on my flight and it would have been unpleasant if he'd ended up sitting next to me.

I was a bit shocked by this incident, though I probably shouldn't have been. This kind of unapologetic rudeness is not something I encounter very often from strangers in face-to-face situations. I guess that means I'm fortunate. Surprisingly I felt quite calm during the conversation and only felt rage later.

Embarrassingly-much-later, it dawned on me that Jesus wants me to forgive this man. I am now making the effort to do so.

Full Article

March 26, 2014 09:46 PM

Introducing rr

Bugs that reproduce intermittently are hard to debug with traditional techniques because single stepping, setting breakpoints, inspecting program state, etc, is all a waste of time if the program execution you're debugging ends up not even exhibiting the bug. Even when you can reproduce a bug consistently, important information such as the addresses of suspect objects is unpredictable from run to run. Given that software developers like me spend the majority of their time finding and fixing bugs (either in new code or existing code), nondeterminism has a major impact on my productivity.

Many, many people have noticed that if we had a way to reliably record program execution and replay it later, with the ability to debug the replay, we could largely tame the nondeterminism problem. This would also allow us to deliberately introduce nondeterminism so tests can explore more of the possible execution space, without impacting debuggability. Many record and replay systems have been built in pursuit of this vision. (I built one myself.) For various reasons these systems have not seen wide adoption. So, a few years ago we at Mozilla started a project to create a new record-and-replay tool that would overcome the obstacles blocking adoption. We call this tool rr.


Here are some of our key design parameters:

  • We focused on debugging Firefox. Firefox is a complex application, so if rr is useful for debugging Firefox, it is very likely to be generally useful. But, for example, we have not concerned ourselves with record and replay of hostile binaries, or highly parallel applications. Nor have we explored novel techniques just for the sake of novelty.
  • We prioritized deployability. For example, we deliberately avoided modifying the OS kernel and even worked hard to eliminate the need for system configuration changes. Of course we ensured rr works on stock hardware.
  • We placed high importance on low run-time overhead. We want rr to be the tool of first resort for debugging. That means you need to start getting results with rr about as quickly as you would if you were using a traditional debugger. (This is where my previous project Chronomancer fell down.)
  • We tried to take a path that would lead to a quick positive payoff with a small development investment. A large speculative project in this area would fall outside Mozilla's resources and mission.
  • Naturally, the tool has to be free software.

I believe we've achieved those goals with our 1.0 release.

There's a lot to say about how rr works, and I'll probably write some followup blog posts about that. In this post I focus on what rr can do.

rr records and replays a Linux process tree, i.e. an initial process and the threads and subprocesses (indirectly) spawned by that process. The replay is exact in the sense that the contents of memory and registers during replay are identical to their values during recording. rr provides a gdbserver interface allowing gdb to debug the state of replayed processes.


Here are performance results for some Firefox testsuite workloads: These represent the ratio of wall-clock run-time of rr recording and replay over the wall-clock run-time of normal execution (except for Octane, where it's the ratio of the normal execution's Octane score over the record/replay Octane scores ... which, of course, are the same). It turns out that reftests are slow under rr because Gecko's current default Linux configuration draws with X11, hence the rendered pixmap of every test and reference page has to be slurped back from the X server over the X socket for comparison, and rr has to record all of that data. So I also show overhead for reftests with Xrender disabled, which causes Gecko to draw locally and avoid the problem.

I should also point out that we stopped focusing on rr performance a while ago because we felt it was already good enough, not because we couldn't make it any faster. It can probably be improved further without much work.

Debugging with rr

The rr project landing page has a screencast illustrating the rr debugging experience. rr lets you use gdb to debug during replay. It's difficult to communicate the feeling of debugging with rr, but if you've ever done something wrong during debugging (e.g. stepped too far) and had that "crap! Now I have to start all over again" sinking feeling --- rr does away with that. Everything you already learned about the execution --- e.g. the addresses of the objects that matter --- remains valid when you start the next replay. Your understanding of the execution increases monotonically.


rr has limitations. Some are inherent to its design, and some are fixable with more work.

  • rr emulates a single-core machine. So, parallel programs incur the slowdown of running on a single core. This is an inherent feature of the design. Practical low-overhead recording in a multicore setting requires hardware support; we hope that if rr becomes popular, it will motivate hardware vendors to add such support.
  • rr cannot record processes that share memory with processes outside the recording tree. This is an inherent feature of the design. rr automatically disables features such as X11 shared memory for recorded processes to avoid this problem.
  • For the same reason, rr tracees cannot use direct-rendering user-space GL drivers. To debug GL code under rr we'll need to find or create a proxy driver that doesn't share memory with the kernel (something like GLX).
  • rr requires a reasonably modern x86 CPU. It depends on certain performance counter features that are not available in older CPUs, or in ARM at all currently. rr works on Intel Ivy Bridge and Sandy Bridge microarchitectures. It doesn't currently work on Haswell and we're investigating how to fix that.
  • rr currently only supports x86 32-bit processes. (Porting to x86-64 should be straightforward but it's quite a lot of work.)
  • rr needs to explicitly support every system call executed by the recorded processes. It already supports a wide range of syscalls --- syscalls used by Firefox --- but running rr on more workloads will likely uncover more syscalls that need to be supported.
  • When used with gdb, rr does not provide the ability to call program functions from the debugger, nor does it provide hardware data breakpoints. These issues can be fixed with future work.


We believe rr is already a useful tool. I like to use it myself to debug Gecko bugs; in fact, it's the first "research" tool I've ever built that I like to use myself. If you debug Firefox at the C/C++ level on Linux you should probably try it out. We would love to have feedback --- or better still, contributions!

If you try to debug other large applications with rr, you will probably encounter rr bugs. Therefore we are not yet recommending rr for general-purpose C/C++ debugging. However, if rr interests you, please consider trying it out and reporting/fixing any bugs that you find.

We hope rr is a useful tool in itself, but we also see it as just a first step. rr+gdb is not the ultimate debugging experience (even if gdb's backtracing features get an rr-based backend, which I hope happens!). We have a lot of ideas for making vast improvements to the debugging experience by building on rr. I hope other people find rr useful as a building block for their ideas too.

I'd like to thank the people who've contributed to rr so far: Albert Noll, Nimrod Partush, Thomas Anderegg and Chris Jones --- and to Mozilla Research, especially Andreas Gal, for supporting this project.

Full Article

March 26, 2014 01:59 AM

March 23, 2014

Robert O'Callahan -- Mozilla And The Silicon Valley Cartel

This article, while overblown (e.g. I don't think "no cold calls" agreements are much of a problem), is an interesting read, especially the court exhibits attached at the end. One interesting fact is that Mozilla does not appear on any of Google's do-not-call lists --- yay us! (I can confirm first-hand that Mozilla people have been relentlessly cold-emailed by Google over the years. They stopped contacting me after I told them not to even bother trying until they open a development office in New Zealand.)

Exhibit 1871 is also very interesting since it appears the trigger that caused Steve Jobs to fly off the handle (which in turn led to the collusion at the heart of the court case) was Ben Goodger referring members of the Safari team for recruitment by Google, at least one of whom was formerly from Mozilla.

Full Article

March 23, 2014 10:42 PM

March 16, 2014

Robert O'Callahan -- Taroko National Park

This weekend, in between the media and layout work weeks, I had time off so I thought I'd get out of Taipei and see a different part of Taiwan. On Alfredo's recommendation I decided to visit Taroko National Park and do some hiking. I stayed at Taroko Lodge and had a great time --- Rihang Su was an excellent host, very friendly and helpful.

On Saturday I took the train from Songshan station in Taipei to Xincheng. From there I went via the lodge to the park's eastern headquarters and did a short afternoon hike up to near Dali village and back down again. This was short in distance --- about 3.5 km each way in two and half hours --- but reasonably hard work since it's about a 900m vertical climb.

On Saturday night Rihang took the lodge guests to Hualien city for dinner at a hotpot buffet place. The food was reasonable and it was a fun dinner. There were instructions in English and in general I was surprised by how much English signage there was in Hualien (including at the supermarket) --- I had expected none.

On Sunday (today) I got up fairly early for the main goal of my trip: Zhuilu Old Trail. Rihang dropped me off at the western end around 8:15am and I walked the 10km length in a bit under four hours. You're supposed to allow for six, but that's probably a bit conservative, plus I'm fairly fast. It is hard work and tricky in places, however. This trail is quite amazing. There's one stretch in particular where the path winds along the cliff face, mostly about a meter wide, and there's nothing between you and a sheer drop of hundreds of meters --- except a plastic-clad steel cable on the cliff side of the track to hold onto. It reminded me of the National Pass track in Australia's Blue Mountains, but more intense. Much of the track has spectacular views of the Taroko Gorge --- sheer cliffs, marble boulders, lush forest, and mountain streams. In one place I was lucky enough to see macaque monkeys, which was thrilling since I've never seen monkeys in the wild before. The trail has an interesting history; it was originally constructed by the Japanese military in the early 20th century (probably using slave labour, though it wasn't clear from the signage), to improve their control over the native people of the area. Later the Japanese turned it into a tourist destination and that's what it is today. Along the trail there are rest areas where fortified "police stations" used to be.

After finishing that trail, since I was ahead of schedule I continued with the Swallow Grotto walk along the road westward up the gorge. This walk was also amazing but offers quite different views since you're near the bottom of the gorge --- and it's much easier. The walk ends at the bottom of the Zhuilu cliff, which is neck-craningly massive. After that I got picked up and went to the train station to head straight back to Taipei.

This was definitely a very good way to see a bit of Taiwan that's not Taipei. Shuttling around between the train station, the lodge, and hiking destinations, I got a glimpse of what life is like in the region --- gardens, farming, stray dogs, decrepit buildings, industry, narrow roads, etc. But the natural beauty of Taroko gorge was definitely the highlight of the trip. I enjoy hiking with friends and family very much, but it's nice to hike alone for a change; I can go at my own pace, not worry about anyone else, and be a bit more relaxed. All in all it was a great weekend getaway.

Full Article

March 16, 2014 04:41 PM


I've been in Taiwan for a week. Last Sunday, almost immediately after checking into the hotel I went with a number of Mozilla people to Maokong Gondala and did a short hike around Maokong itself, to Yinhe Cave and back. I really enjoyed this trip. The gondala ride is quite long, and pretty. Maokong itself is devoted to the cultivation and vending of tea. I haven't seen a real tea plantation before, and I also got to see rice paddies up close, so this was quite interesting --- I'm fascinated by agricultural culture, which dominated so many people's lives for such a long time. Yinhe Cave is a cave behind a waterfall which has been fitted out as a Buddhist temple; quite an amazing spot. We capped off the afternoon by having dinner at a tea house in Maokong. A lot of the food was tea-themed --- deep-fried tea leaves, tea-flavoured omelette, etc. It was excellent, a really great destination for visitors to Taipei.

Full Article

March 16, 2014 10:12 AM

March 10, 2014

Robert O'Callahan -- Introducing Chaos Mode

Some test failures are hard to reproduce. This is often because code (either tests or implementation code) makes unwarranted assumptions about the environment, assumptions that are violated nondeterministically. For example, a lot of tests have used setTimeout to schedule test code and assumed certain events will have happened before the timeout, which may not be true depending on effects such as network speeds and system load.

One way to make such bugs easier to reproduce is to intentionally exercise nondeterminism up to the limits of API contracts. For example, we can intentionally vary the actual time at which timers fire, to simulate the skew between CPU execution time and real time. To simulate different permitted thread schedules, we can assign random priorities to threads. Since hashtable iteration is not defined to have any particular order, we can make a hashtable iterator always start at a randomly chosen item.

I tried applying this to Gecko. I have patches that define a global "chaos mode" switch, and in several different places, if we're in chaos mode, we choose randomly between different valid behaviors of the code. Here's what the patches currently do:

  • Sometimes yield just before dispatching an XPCOM event. This gives another thread a chance to win an event-dispatch race.
  • On Linux, give threads a random priority and pin some threads to CPU 0 so they contend for CPU.
  • Insert sockets in random positions in the list of polled sockets, to effectively randomize the priority of sockets in poll results.
  • Similarly, when putting HTTP transactions into the HTTP transaction queue, randomly order them among other transactions with the same specified priority.
  • Start hashtable iteration at a random entry.
  • Scale timer firing times by random amounts (but don't vary the order in which timers fire, since that would violate the API contract).
  • Shuffle mochitests and reftests so they run in random order.

Note that it can be valuable to make a single random choice consistently (for the same object, thread, etc) rather than making lots of fine-grained random decisions. For example, giving a thread a fixed low priority will starve it of CPU which will likely cause more extreme behavior --- hopefully more buggy behavior --- than choosing a random thread to run in each time quantum.

One important source of nondeterminism in Gecko is XPCOM event (i.e. HTML5 task) dispatch. A lot of intermittent bugs are due to event timing and ordering. It would be nice to exploit this in chaos mode, e.g. by choosing the next event to fire randomly from the set of pending events instead of processing them in dispatch order. Unfortunately we can't do that because a lot of code depend on the API contract that firing order follows dispatch order. In general it's hard to determine what the valid alternative firing orders are; the first item on my list above is my best approximation at the moment.

Important Questions

Does this find bugs? Yes:

Which chaos features are the most helpful for producing test failures? I don't know. It would be a very interesting experiment to do try pushes with different patches enabled to figure out which ones are the most important.

Does it help reproduce known intermittent bugs? Sometimes. In bug 975931 there was an intermittent reftest failure I could not reproduce locally without chaos mode, but I could reproduce with chaos mode. On the other hand chaos mode did not help reproduce bug 791480. Extending chaos mode can improve this situation.

Isn't this just fault injection? It's similar to fault injection (e.g. random out-of-memory triggering) but different. With fault injection typically we expect most tests to fail because faults like OOM are not fully recoverable. Chaos mode should not affect any correctness properties of the program.

Wasn't this already done by <insert project name>? Probably. I don't claim this is a new idea.

When is this going to land and how do I turn it on? It has already landed. To turn it on, change isActive() to return true in mfbt/ChaosMode.h. Shuffling of reftests and mochitests has to be done separately.

OK, so this can trigger interesting bugs, but how do we debug them? Indeed, chaos mode makes normal debugging workflows worse by introducing more nondeterminism. We could try to modify chaos mode to reproduce the random number stream between runs but that's inadequate because other sources of nondeterminism would interfere with the order in which the random number stream is sampled. But we are working on a much better solution to debugging nondeterministic programs; I'll be saying more about that very soon!

Full Article

March 10, 2014 12:46 AM

March 09, 2014

Robert O'Callahan -- My Linkedin Account Is Dead, And Why Is Google Being Stupid?

I tried to delete my Linkedin account a while back, but I still get a lot of "invitation to connect on Linkedin" emails. I plan to never connect to anyone on Linkedin ever again, so whoever wants to connect, please don't be offended when it doesn't happen --- it's not about you.

Linkedin, I despise you for your persistent spam.

PS, I'm visiting Taiwan at the moment and wondering why Google uses that as a cue to switch its Web interface to Chinese even when I'm logged into my regular Google account. Dear Google, surely it is not very likely that my change of location to Taiwan indicates I have suddenly learned Chinese and forgotten English.

Full Article

March 09, 2014 11:56 PM

Last updated: November 17, 2014 07:00 AM