Planet Mozilla Education

Open Badges started as a modest experiment: build open source badge issuing software for ourselves and others. As momentum around this experiment has grown, it feels like the opportunity is bigger: we could build openness and user empowerment into how learning — and professional identity — work all across the web. With Open Badges 1.0 out there in the world, now is the right time to ask: where next for Mozilla and badges?

When Mozilla and MacArthur Foundation first started work on Open Badges about 18 months ago, the plan was to build a badge interchange standard (like SMTP for skills) and a collection of open source software for issuing and sharing badges (Badge Backpack, Open Badger, etc.). We’ve built all these things. And we’ve put up a reference implementation that Mozilla and others are using. This was really the limit of our original plan: build some basic open tech for badges and put it out there in the world.

The thing is: there has been way more excitement and pick up of badges than we expected. Even though Open Badges only launched officially in March, there are already over 800 unique providers who have issued almost 100,000 badges. We are also starting to see the development of city-wide systems where learners can pick up hundreds of different badges from across dozens of learning orgs and combine them all into a single profile. Chicago is the first city to do this (June 1), but Philadelphia and San Francisco are not far behind. And, this is just the tip of the iceberg: orgs like the Clinton Global Initiative and the National Science Foundation are focusing on badges in a way that is likely to drive even more educators to pick up the Open Badges standard, making their badges interoperable with others.

Of course, the fact that educators and policy makers are interested in badges doesn’t represent a victory in itself. It just shows momentum and buzz. The real opportunity — and the real impact — comes when learners and employers get excited about badges. Mozilla never planned to build offerings for these audiences. Increasingly, it feels like we should.

In the Internet era, people learn things online and out of school all the time. Whether you want to make a web page, knit a sweater or get better at calculus, the internet makes it easy to learn on your own or with a group of friends outside of a school setting. However, there is no good way to get credentials or recognition for this kind of learning. And, even if there was, there is no trusted, verifiable way to plug that recognition into Facebook, About.me and other places that make up your online identity. People have no good way to show ‘what they know’ online.

Similarly, employers are increasingly turning to the internet to find talent. They use sites like LinkedIn that let you search online resumes. Or, increasingly, to sites like Gild and TalentBin that use data mining to find potential hires. The problem: these services do not offer granular or variable skills profiles. And, with some of them, there are significant issues around privacy: people are being offered up as potential hires without even knowing that these sites are collecting data about them.

Mozilla could offer a distributed, open source and privacy-friendly solution to problems like these. We could help learners show their skills in all their online profiles and also help employers search for talent reliably. However, to do so, we’d have to build a Firefox-quality offering for learners and employers on top of Open Badges. While this hasn’t been our focus up til now, I’m thinking more and more that this is something we should consider.

In some ways, there is a parallel to Gecko and Firefox. Gecko provides the underlying platform for shaping standards around our vision of the web. But we need a popular consumer offering like Firefox if we want this vision to actually become relevant in the market. Right now, with Open Badges, we’re mostly just playing at the underlying standards layer. If we really want to shape how learning and professional identity work on the web, we probably need to build our own offerings directly for the people who most want and need badges.

Now is the time to be looking at where the opportunity is in this space. Momentum and demand is amongst educators is growing. More and more start ups are appearing in the badges, portfolio and skills spaces. And likelihood that badges will be important for learners and employers is growing. We need to be asking ourselves: how can Mozilla — and its values — shape this space?

With this in mind, Erin Knight is leading an effort over the next few months to look at different badges product options. She’ll be providing updates on her blog. And I’ll be summarizing here as well. If you have ideas on where Mozilla should go on all of this, we’d love to have you involved as we think this through. Comments here on this post are a good place to start.

May 20, 2013 11:22 AM

I knew Dennis Lane only slightly: I occasionally commented on his blog, he commented on mine once or twice, and I met and talked to him several times at Howard County blogger meetups and other events. I can’t speak to his life as a private person and how he came to a violent end, and even if I could I wouldn’t: I don’t blog about my own private and family life, and won’t do so about others. However I did want to say a few words to mark his death.

What I admired most about Dennis as a blogger was his ability to write frequently and seemingly extemporaneously on a variety of topics: local politics, real estate and business affairs, and just the normal goings-on of daily life. He mentioned recently that he hadn’t had time to blog as much as normal, but (as I commented on the post), even his infrequent blogging way outpaced my infrequent offerings. I also enjoyed reading his Business Monthly columns, whenever I picked up a copy at a local establishment; it was one of the few things I read on paper rather than online. Leaving aside all his other contributions to Howard County, he’ll be sorely missed in the local blogging community, of which he was a founder and guiding light.

A couple of other points: In 2012 four people were murdered in Howard County and I was acquainted with two of them, Mary-Marguerite Kohn and Brenda Brewington, who were shot at St Peters Episcopal Church in Ellicott City almost exactly a year ago. This year I think there have been two homicides in Howard County thus far, and I’m acquainted with one of the victims. Strange that in a placid suburban county of 300,000 violent death should so disproportionately strike people I know.

FInally, it’s fitting that I first learned of Dennis’s death while reading a blog post, in this case by TJ Mayotte. It’s a measure of how much I rely on local blogs and hyper-local sites like Ellicott City Patch and Explore Howard for my knowledge of Howard County affairs. I guess one way to honor Dennis and what he meant to our local online community is to get off my rear end and blog some more myself.

May 10, 2013 11:49 PM

At a high level, the core of the comm-central build system is not a single build system but rather three copies of the same build system. In simple terms, there's a matrix on two accesses: which repository does the configuration of code (whose config.status invokes it), and which repository does the build (whose rules.mk is used). Most code is configured and built by mozilla-central. That comm-central code which is linked into libxul is configured by mozilla-central but built by comm-central. tier_app is configured and built by comm-central. This matrix of possibilities causes interesting bugs—like the bustage caused by the XPIDLSRCS migration, or issues I'm facing working with xpcshell manifests—but it is probably possible to make all code get configured by mozilla-central and eliminate several issues for once and all.

With that in mind, here is a step-by-step look at how the amorphous monster that is the comm-central build system works:

python client.py checkout

And comm-central starts with a step that is unknown in mozilla-central. Back when everyone was in CVS, the process of building started with "check out client.mk from the server, set up your mozconfig, and then run make -f client.mk checkout." The checkout would download exactly the directories needed to build the program you were trying to build. When mozilla-central moved to Mercurial, the only external projects in the tree that Firefox used were NSPR and NSS, both of which were set up to pull from a specific revision. The decision was made to import NSPR and NSS as snapshots on a regular basis, so there was no need for the everyday user to use this feature. Thunderbird, on the other hand, pulled in the LDAP code externally, as well as mozilla-central, while SeaMonkey also pulls in the DOM inspector, Venkman, and Chatzilla as extensions. Importing a snapshot was not a tenable option for mozilla-central, as it updates at an aggressive rate, so the functionality of checkout was ported to comm-central in a replacement python fashion.

./configure [comm-central]

The general purpose of configure is to discover the build system and enable or disable components based on options specified by the user. This results in a long list of variables which is read in by the build system. Recently, I changed the script to eliminate the need to port changes from mozilla-central. Instead, this script reads in a few variables and tweaks them slightly to produce a modified command line to call mozilla-central's configure...

./configure [mozilla-central]

... which does all the hard work. There are hooks in the configure script here to run a few extra commands for comm-central's need (primarily adding a few variables and configuring LDAP). This is done by running a bit of m4 over another file and invoking that as a shell script; the m4 is largely to make it look and feel "like" autoconf. At the end of the line, this dumps out all of the variables to a file called config.status; how these get configured in the script is not interesting.

./config.status [mozilla/comm-central]

But config.status is. At this point, we enter the mozbuild world and things become very insane; failure to appreciate what goes on here is a very good way to cause extended bustage for comm-central. The mozbuild code essentially starts at a directory and exhaustively walks it to build a map of all the code. One of the tasks of comm-central's configure is to alert mozbuild to the fact that some of our files use a different build system. We, however, also carefully hide some of our files from mozbuild, so we run another copy of config.status again to add in some more files (tier_app, in short). This results in our code having two different notions of how our build system is split, and was never designed that way. Originally, mozilla-central had no knowledge of the existence of comm-central, but some changes made in the Firefox 4 timeframe suddenly required Thunderbird and SeaMonkey to link all of the mailnews code into libxul, which forced this contorted process to come about.

make

Now that all of the Makefiles have bee generated, building can begin. The first directory entered is the top of comm-central, which proceeds to immediately make all of mozilla-central. How mozilla-central builds itself is perhaps an interesting discussion, but not for this article. The important part is that partway through building, mozilla-central will be told to make ../mailnews (or one of the other few directories). Under recursive make, the only way to "tell" which build system is being used is by the directory that the $(DEPTH) variable is pointing to, since $(DEPTH)/config/config.mk and $(DEPTH)/config/rules/mk are the files included to get the necessary rules. Since mozbuild was informed very early on that comm-central is special, the variables it points to in comm-central are different from those in mozilla-central—and thus comm-central's files are all invariably built with the comm-central build system despite being built from mozilla-central.

However, this is not true of all files. Some of the files, like the chat directory are never mentioned to mozilla-central. Thus, after the comm-central top-level build completes building mozilla-central, it proceeds to do a full build under what it thinks is the complete build system. It is here that later hacks to get things like xpcshell tests working correctly are done. Glossed over in this discussion is the fun process of tiers and other dependency voodoo tricks for a recursive make.

The future

With all of the changes going on, this guide is going to become obsolete quickly. I'm experimenting with eliminating one of our three build system clones by making all comm-central code get configured by mozilla-central, so that mozbuild gets a truly global view of what's going on—which would help not break comm-central for things like eliminating the master xpcshell manifest, or compiling IDLs in parallel. The long-term goal, of course, is to eliminate the ersatz comm-central build system altogether, although the final setup of how that build system works out is still not fully clear, as I'm still in the phase of "get it working when I symlink everything everywhere."

May 08, 2013 01:46 AM

Plans are coming together for Mozilla’s Maker Party 2013. And I’m getting excited. Last year’s party had people making things on the web at 700 local events in 80 countries. This year it’ll be bigger. But, more important, I think this year will plant the foundations for something that lasts well beyond the campaign: a movement of people who want to teach 10s of millions of people how the web works.

Mozilla has built this kind of movement before: when we first launched Firefox. Many people just downloaded Firefox 1.0 because it was great. But others became on-the-ground evangelists and promoters. They told their friends about Firefox. They installed it on other people’s computers. They showed them how to use bookmarks, and pop-up blockers and add-ons. And, over 10,000 of them of them put up money to tell the world about Firefox in a historic two-page Sunday New York Times ad.

In my view, the mentors and local champions who will step up to organize the Mozilla Maker Party are just like the early enthusiasts who helped Firefox get to 500 million users. It’s these people who will show the first million Webmakers what they can make. Who will start awarding badges that reward people for their skill and creativity on the web. And who will create excitement about all the tools and programs across the web the empower people to make and create. These mentors and local champions are the core leaders that Mozilla needs if we want to teach the world the web.

Building on last year’s successful Summer Code Party, Maker Party 2013 has a number of pieces designed specifically to help mentors and local champions succeed. Five that I’m really excited about are:

1. Teach the Web: a nine-week free and open online course for people who want to be Mozilla mentors and local champions. It’s highly collaborative, convening nearly 3,000 participants to share their teaching practice, learning materials and learn to hack the web on the way. The course started last week, but you can still sign up here www.webmaker.org/teach

2. Super mentors: these are the passionate volunteers who really make the online course and marquee Maker Parties happen. They are experienced in teaching the web, running events and creating teaching materials. Starting with their work on Teach the Web, the Super Mentors are the leadership core of the larger Webmaker Mentor community. We already have over 100 super mentors. We hope to have many more by the time Maker Party 2013 is done.

3. A big tent with more than 40 partner organizations joining the Maker Party and carrying out making-and-learning activities across the globe. Like Mozilla, these organizations are part of a growing movement to teach the web and promote the maker spirit with hands-on learning. This network of partners is critical to growing this movement: there is no way any one organization can do this on its own. Mentors can bring their own organizations into this tent as a way to get publicity and recognition, or just as a way to be part of the party.

4. Hackable Activity Kits: simple ‘instructables’ that you can use show people how to make web pages, Popcorn videos, etc. The guides are hackable, forkable HTML pages so you can customize them. OpenMatt explains these kits well in this post.

5. An improved webmaker.org: We’re launching some new features on webmaker.org June 15 to designed for making and learning on the web. Not only have these tools have been designed with mentors in mind, we’ll also be taking mentor feedback and improving them on a constant basis.

While the Maker Party campaign runs from June to September, Mozilla’s hope is to build a lasting network of people around the world who want to teach people how the web works. In September, we’ll be inviting mentors and local organizers to stay involved in Webmaker. This will include invites to MozFest 2013 in London this October, opportunities for continued online mentoring and local organizing and a chance to help shape where we take the Webmaker mentor program in 2014+. In many ways, Maker Party is a kick off for these lasting activities.

If you are someone who wants to teach the world how the web works — or even just show a few people how to get more creative online — you should get involved. You can start by joining the Teach the Web course or just signing up for Maker Party 2013 updates. Also, start thinking about what you might want to do in your town or city in the coming months. Getting people excited about the web is actually pretty easy. And fun.

May 07, 2013 03:38 PM

A better picture of Webmaker v2 has snapped into focus over the past few weeks. The current plan builds on the ‘Webmaker as a popular way to make and learn on the web’ vision we set out in December. What’s clearer now is our focus on people who already take photos, blog and create online: we give them new ways to make, remix. and improve their craft. We also them access to mentors committed to helping others learn how the web works.

In this post, I wanted to pull out my top 5 list of things I’m excited about in Webmaker v2:

 1. Rebooting the brand to focus on makers of all ages

Cassie, Kate, Chris and others already working to reboot the Webmaker brand and UX to really emanate the maker spirit.

The idea is to appeal to teens and above, not kids. Also, to target people who already ‘make’ in some sense. You can see hints of this in their early mockups.

2. Building a gallery to show all the awesome makes

The biggest gap in Webmaker v1 was the lack of a gallery where you can see what people made. Fixing this is the top priority for Webmaker v2.

The site will lead with tiles of the best things people have made. More importantly, the site will be filled with all sorts of different galleries: makes that teach you how to make a similar thing; makes you made; makes on specific themes; makes that are actually curriculum materials.

3. Creating a Make API so anyone can make a gallery

In related news: we’ve started work on a ‘Make API’ that will let anyone pull a slice of Webmaker content to create their own gallery or service.

At the simplest level, this is a win as it gives us a common publishing model for both Thimble and Popcorn. But, in the long run, the Make API could be something more radical: it’s way for people to store, describe, slice, dice and share any blob of HTML from across the web. Ultimately, it could help people to take control of all the things they make online, no matter where they’ve made them.

 4. Deepening learning w/ challenges + badges

Webmaker v2 will include peer reviewed badges based where: 1) we describe a skill; 2) someone submits something they made that demonstrates that skill; 3) a peer or mentor reviews the submission and awards the badge (or not).

This is exciting because a) we can badge for skills defined in the Mozilla web literacy standard and b) people can submit ‘makes’ made with any tool (e.g. Scratch). This second piece is essential if we want to open things up widely on the making as learning front: people don’t just want to make things with Popcorn and Thimble.

5. Making it easy to make hacktivity kits using Thimble

For Webmaker to succeed, we need any mentor in our network to be able to write or remix Hackable Activity Kits.

Currently, that’s difficult as our learning materials are all hard coded web pages that need someone with commit privileges to check in. We’re going to change this by making it possible to create these kits directly in Thimble and then creating a special gallery for these pages. The result: a constantly updated community run gallery of learning materials.

–

These 5 things — and everything in the Webmaker v2 product vision — represent a big leap forward. When we started 2013, we had a fragmented offering with no single sign on, no gallery and no publishing model. We’re moving to a place where we not only have a unified offering but also something that is flexible in terms of how people publish and how they learn. New features and improvements will roll out weekly over the course of the summer, starting June 15. If you want to track progress on Webmaker v2, follow this scrumbug and this Tumblr blog.

April 25, 2013 01:52 PM

Today’s graph is a scatter plot of out-of-memory crashes. It categorizes crashes according to the smallest block of available VM and the amount of available pagefile space.

There were roughly 1000 crashes due to bug 829954 between 10-April and 15-April 2013. Click on individual crash plots to see memory details and a link to the crash report.

Direct link to SVG file. Link to raw data.

Conclusions

After graphing these crashes, it seems clear that there are two distinct issues:

• Crashes which are above the blue line and to the left have free space in their page file, but we have run out of contiguous virtual memory space. This is likely caused by the virtual memory leak from last week.
• Crashes which are below the blue line and to the right have available virtual memory, but don’t have any real memory for allocation. It is likely that the computer is already thrashing pretty heavily and appears very slow to the user.

I was surprised to learn that not all of these crashes were caused by the VM leak.

The short-term solution for this issue remains the same: the Mozilla graphics engine should stop using the infallible/aborting allocator for graphics buffers. All large allocations (network and graphics buffers) should use the fallible allocator and take extra effort to be OOM-safe.

Long-term, we need Firefox to be aware of the OS memory situation and continue to work on memory-shrinking behavior when the system starts paging or running out of memory. This includes obvious behaviors like throwing away the in-memory network and graphics caches, but it may also require drastic measures such as throwing away the contents of inactive tabs and reloading them later.

Charting Technique

With this post, I am starting to use a different charting technique. Previously, I was using the Flot JS library to generate graphs. Flot makes pretty graphs (although it doesn’t support labeling Axes without a plugin!). It also features a wide range of plugins which add missing features. But often, it doesn’t do exactly what I want and I’ve had to dig deep into its guts to get things to look good. It is also cumbersome to include dynamically generated JS graphs in a blog post, and the prior graphs have been screenshots.

This time around, I generated the graph as an SVG image using the svgwrite python library. This allows me to put the full SVG graph directly into the blog, and it also allows me to dynamic features such as rollovers directly in these blog posts. Currently I’m setting up the axes and labels manually in python, but I expect that this will turn into a library pretty quickly. I experimented with svgplotlib but the installation requirements were too complex for my needs.

I’m not sure whether or not the embedded SVG will make it through feed aggregators./readers or not. Leave comments if you see weird results.

April 24, 2013 07:55 PM

Sometimes I make a graph to confirm a theory. Sometimes it doesn’t work. This is one of those days.

I created this graph in an attempt to analyze bug 837835. In that bug, we are investigating an increase in the number of crash reports we receive which have an empty (0-byte) minidump file. We’re pretty sure that this usually happens because of an out-of-memory condition (or an out of VM space condition).

Robert Kaiser reported in the bug that he suspected two date ranges of causing the number of empty dumps to increase. Those numbers were generated by counting crashes per build date. But they were very noisy, partly because they didn’t account for the differences in user population between nightly builds.

In this graph, I attempt to account for crashes per user. This was a slightly complicated task, because it assembles information from three separate inputs:

• ADU (Active Daily Users) data is collected by Metrics. After normalizing the data, it is saved into the crash-stats raw_adu table.
• Build data is pulled into the crash-stats database by using a tool called ftpscraper and saved into the releases_raw table. Anything called “scraper” is finicky and changes to other system can break it.
• Crash data is collected directly in crash-stats and stored in the reports_clean table.

Unfortunately, each of these systems has their own way of representing build IDs, channel information, and operating systems:

In this case, I’m only interested in the Windows data, and we can safely assuming that almost all of the empty minidump crashes occur on Windows. The script/SQL query to collect the data simply limits each data source separately and then combines them after they have been limited to windows nightly builds, users, and crashes.

This missing builds are the result of ftpscraper failure.

I’m not sure what to make of this data. It seems likely that we may have fixed part of the problem in the 2013-01-25-03-10-18 nightly. But I don’t see a distinct regression range within this time frame. Perhaps around 25-December? Of course, it could also be that the dataset is so noisy that we can’t draw any useful conclusions from it.

April 22, 2013 07:40 PM

At last week’s Mozilla Foundation board meeting, we looked at what we’ve done so far in 2013 and what we need to do next. Key messages from the discussion: We’re making good progress on Webmaker. We we shipped better Popcorn and Badges tools. We added a ‘teach’ section to webmaker.org. We undertook experiments with new kinds of remixable content.

However, we still need to roll all this into a Webmaker v2 that will excite and provide value to makers. Also, we need to recognize that we’re doing more than just Webmaker this year: Open Badges is growing even more rapidly than expected. I’ve posted slides from the board meeting here and summarized the content below

As a reminder: our overarching goal for 2013 is to turn Webmaker in a popular way to both make and learn on the web. We set these more specific goals:

• Goal #1: make Webmaker into popular way to make and remix web content (target = 250k makers)
• Goal #2: build better ways to level up skills, craft and code as you make (target = 1M badges)
• Goal #3: grow our global community of mentors to power Webmaker (target = 10k mentors)

While we didn’t explicitly make it a top level goal, it’s clear that ‘make Open Badges successful’ and ‘respond to the demand we’re seeing for badges’ have also become major priorities for 2013.

We’ve made a solid start on all these goals in Q1: building the foundations for Webmaker v2 and growing the Open badges project significantly. Some highlights re: things we shipped and balls we moved:

• PopcornMaker shipped ability to use multiple media files plus better social media sharing.
• Open Badges / Badger shipped 1.0, now ready for Webmaker plus 700 other partner sites.
• Webmaker.org added a ‘teach’ section to house hacktivity kits and other resources for mentors.
• Using Hive and ReMo as a base, we’ve built a core cadre of Webmaker ‘super mentors’.
• Maker Party 2013 is teed up as a major platform for Mozilla and 40+ partner orgs.

While this is solid progress, it’s important to recognize that we still need to roll all of this into a Webmaker v2 that will truly excite and provide value to makers, mentors and learners. Challenges we face include:

• We’ve had a tough time finding right mix of making (goal #1) and learning (goal #2) in the remixable content we’ve been developing for Webmaker v2. We need to get better at adding elements designed to teach specific webmaking skills..
• We started the year with a fragmented technical infrastructure: we’ve had to invest a lot of time in underlying technology like single sign on and a shared publishing system for all Webmaker content.
• We still don’t have an approach to localization: this hard to nail down given the fact that our tools and site are going through rapid changes.

Another challenge is that the scope of our goals is shifting: Open Badges and Open News continue to grow as major initiatives above an beyond what we’re doing with Webmaker. We need to accept the fact that we’re still a multi product / project org and find a way to better support this growth.

The good news: we have a clear plans in place that aim directly at these challenges. The top three priorities as we move through Q2 are:

• Priority #1: ship a v2 of webmaker.org that attracts makers and bakes in learning. (Lead: Brett, w/ Paula)
• Priority #2: drive momentum with mentors and Maker Party 2013, act like a movement. (Lead: Chris, w/ Erica)
• Priority #3: strong Open Badges proof point via Chicago Summer of Learning. (Lead: Erin)

Work on all three of these priorities is well underway and we are making good progress. As we do this work, there a three questions we should be actively discussing:

• Do we have the right making and learning balance as Webmaker v2 ships?
• Are we providing value to makers and mentors? Are they using the site? Running events?
• How can we support Open Badges to respond to growing demand? (June board meeting)

We should all be keeping these things in mind as we build out Webmaker v2, Maker Party and Chicago Summer of Learning, especially the question: are we providing value?

It’s an interesting and intense time. Real traction on our big dreams is within sight: a Mozilla-backed movement where people champion creativity and making on the web; a new era of remixable, Legolike web content; a world of learning that works like the web. At the same time, we’re all heads down on the details of building tools, shipping web sites, making content, writing curriculum and recruiting partners. While it can be stressful, this its actually a very good, Mozilla-like place to be. Our hands are mucky shipping things while we are still aimed at and inspired by big dreams of making the web a better place.

Over the next few months, its going to be important to help each other keep this balance. Reach out to someone working on another part of the project to understand what they are working on. Pitch in as people test and irritate what they’re building. Offer advice to new community members as they show up for the first time (thats going to start happening slot). It may feel like we’re all working on different things: but everything we’re doing all points in the same direction of inspiring and empowering people using the web.

April 22, 2013 01:12 PM

I’m working across a lot of different repos these days, and while fixing a whitespace issue in one of my patches, I wondered if it would be possible to eliminate this altogether.

Working with git, we already have a version of what I want in terms of core.autocrlf.  You wrote this patch on Windows, I’m on OS X, and our server is Linux?  No problem, we’ll just use Unix EOLs and have git automatically manage the line endings whenever we checkout or commit.  In other words, when I checkout the code locally, I get a version of it customized to my environment.  There’s a canonical version that git will store, but every developer can have something different in their editor.

Now let’s go one step further.  Imagine you need to add the following code, as I did today:

module.exports = function( req, res ) {
  res.json({
    http: "okay"
  });
};

How many arguments can you have about how to style that code?  Come on, what would you r- in this?  “Dave, no space between `(` and `req` please;” “Dave, you need to indent with 4-spaces not 2;” “Dave, you should just put that `res.json` all on one line.”

The truth is, there’s no right answer here.  Of course there are famous examples of automatic semi-colon insertion causing havoc (`return\n{` vs. `return {`), but let’s stick with my example code for a minute.  Could we let git manage the whitespace here?  Could we, in other words, take that code above, and when I commit, actually store this:

module.exports=function(req,res){res.json({http:"okay"});};

Now imagine that I’ve got a .gitstyle file in my tree, or I’ve somehow specified config options globally, and I can tell git how to style this code again when I check it out.  Imagine you like 8-space indents and I like 2-spaces; imagine you’re allergic to spaces around arguments, and I love them; imagine your OCD demands that all variable declarations line up on the equal sign, and I’m happy to let them wind their way down my editor like a raindrop on a window.  And, imagine we can both do what we like with the same code in the same tree!  Imagine there’s no style guide, it’s easy if you try…

BUT DAVE, THIS CAN’T WORK BECAUSE <strong-reason />!!!  Sure, there’s all sorts of things to work out.  This is a thought experiment.  I’m supposed to be grading final assignments right now, and I needed to procrastinate.  But if this could work, it would be so, so useful!

Patches accepted (please use 2-space indents).

April 20, 2013 06:55 PM

Today’s graph charts the percentage of Firefox users who have known-insecure versions of Flash. It also allows us to visually see the impact of various plugin blocks that have been staged over the past few months.

We are gradually rolling out blocks for more and more versions of Flash. In order to make sure that the blocklist was not causing significant user pain, we started out with the oldest versions of Flash that have the fewest users. We have since been expanding the block to include more recent versions of Flash that are still insecure. We hope to extend these blocks to all insecure versions of Flash in the next few months.

From the data, we see that users on very old versions of Flash (Flash 10.2 and earlier) are not changing their behavior because of the blocklist. This either means that the users never see Flash content, or that they always click through the warning. It is also possible that they attempted to upgrade but for some reason are unable.

Users with slightly newer versions seem more likely to upgrade. Over about a month, almost half of the users who had insecure versions of Flash 10.3-11.2 have upgraded.

Finally, it is interesting that these percentages drop down on the weekends. This indicates that work or school computers are more likely to have insecure versions of Flash than home computers. Because there are well-known exploits for all of these Flash versions, this represents a significant risk to organizations who are not keeping up with security updates!

View the chart in HTML version and the raw data. This data was brought to you by Telemetry, and so the standard cautions apply: telemetry is an opt-in sample on the beta/release channels, and may under-represent certain populations, especially enterprise deployments which may lock telemetry off by default. This data represents Windows users only, because we just recently started collecting Flash version information on Mac, and the Linux Flash player doesn’t expose its version at all.

Raw aggregates for Flash usage can be found in my dated directories on crash-analysis.mozilla.com, for example yesterday’s aggregate counts. You are welcome to scrape this data if you want to play with it; I am also willing to provide interested researchers with additional data dumps on request.

April 19, 2013 03:04 PM

In general, people who are running the Firefox Nightly and Aurora channel are offered a new build every day. But users don’t update immediately, because Firefox does not interrupt you with an update prompt upon receiving an update. Instead it waits and applies the update at the next Firefox restart, or prompts the user to update only after significant idle time.

This means that there is a noticeable “delay” between a nightly build and when people start reporting bugs or crashes against the build. It also means that the number of users using any particular nightly build can vary widely. The following charts demonstrate this variability and the update adoption curves:

Because of this variability, engineers and QA should use care when using data from nightly builds. Note the following conclusions and recommendations:

• Holidays, weekends, and other unexplained factors may mean that some nightly builds get below-average user totals.
• Users often skip nightlies: reported regression ranges should be verified.
• Reliable crash metrics will not be available for several days after a nightly build is released.
• It may be necessary to correlate crash rates on particular builds against the user counts for that build in order to accurately measure crashes-per-user.
• When multiple nightlies are built on the same day (for example, a respin for a bad regression), the user count for each build will be lower than an average nightly build.

This data was collected from ADU data provided by metrics and mirrored in the crash-stats database. The script used to collect this data is available in socorro-toolbox.

April 15, 2013 07:09 PM

When Firefox crashes, it submits a minidump of the crash; it also submits some key/value metadata. The metadata includes basic information about the product/version/buildid, but it also includes a bunch of other information that we use to group, correlate, and diagnose crashes. Using data collected with jydoop, I created a graph of how frequently various metadata keys were submitted on various platforms:

The jydoop script to collect this data is simple:

import crashstatsutils
import json
import jydoop

setupjob = crashstatsutils.dosetupjob([('meta_data', 'json'), ('processed_data', 'json')])

def map(k, meta_data, processed_data, context):
    if processed_data is None:
        return

    try:
        meta = json.loads(meta_data)
        processed = json.loads(processed_data)
    except:
        context.write('jsonerror', 1)
        return

    product = meta['ProductName']

    os = processed.get('os_name', None)
    if os is None:
        return

    context.write((product, os, '$total'), 1)
    for key in meta:
        context.write((product, os, key), 1)

combine = jydoop.sumreducer
reduce = jydoop.sumreducer

View the raw data (CSV).

If you are interested in learning what each piece of metadata means and is used for, I’ve started to document them on the Mozilla docs site.

This graph was generated using the flot JS library and the code is available on github.

April 12, 2013 05:31 PM

I spend a lot of time making sense out of data, and so I’m going to try a new “Graph of the Day” series on this blog.

Today’s plot was created from a crash report submitted by a Firefox user and filed in bugzilla. This user had been experiencing problems where Firefox would, after some time, start drawing black boxes instead of normal content and soon after would crash. Most of the time, his crash report would contain an empty (0-byte) minidump file. In our experience, 0-byte minidumps are usually caused by low-memory conditions causing crashes. But the statistics metadata reported along with the crash show that there was lots of available memory on the system.

This piqued my interest, and fortunately, at least one of the crash reports did contain a valid minidump. Not only did this point us to a Firefox bug where we are aborting when large allocations fail, but it also gave me information about the virtual memory space of the process when it crashed.

When creating a Windows minidump, Firefox calls the MinidumpWriteDump function with the MiniDumpWithFullMemoryInfo flag. This causes the minidump to contain a MINIDUMP_MEMORY_INFO_LIST block, which includes information about every single block of memory pages in the process, the allocation base/size, the free/reserved/committed state, whether the page is private (allocated) memory or some kind of mapped/shared memory, and whether the page is readable/writable/copy-on-write/executable.

(view the plot in a new window).

There are two interesting things that I learned while creating this plot and sharing it on mozilla.dev.platform:

Virtual Memory Fragmentation

Some code is fragmenting the page space with one-page allocations. On Windows, a page is a 4k block, but page space is not allocated in one-page chunks. Instead, the minimum allocation block is 16 pages (64k). So if any code is calling VirtualAlloc with just 4k, it is wasting 16 pages of memory space. Note that this doesn’t waste memory, it only wastes VM space, so it won’t show up on any traditional metrics such as “private bytes”.

Leaking Memory Mappings

Something is leaking memory mappings. Looking at the high end of memory space (bottom of the graphical plot), hover over the large blocks of purple (committed) memory and note that there are many allocations that are roughly identical:

• Size: 0×880000
• State: MEM_COMMIT
• Protection: PAGE_READWRITE PAGE_WRITECOMBINE
• Type: MEM_MAPPED

Given the other memory statistics from the crash report, it appears that these blocks are actually all mapping the same file or piece of shared memory. And it seems likely that there is a bug somewhere in code which is mapping the same memory repeatedly (with MapViewOfFile) and forgetting to call UnmapViewOfFile when it is done.

Conclusion

We’re still working on diagnosing this problem. The user who originally reported this issue notes that if he switches his laptop to use his integrated graphics card instead of his nvidia graphics, then the problem disappears. So we suspect something in the graphics subsystem, but we’re not sure whether the problem is in Firefox accelerated drawing code, the Windows D3D libraries, or in the nvidia driver itself. We are looking at the possibility of hooking allocations functions such as VirtualAlloc and MapViewOfFile in order to find the call stack at the point of allocation to help determine exactly what code is responsible. If you have any tips or want to follow along, see bug 859955.

April 11, 2013 04:43 PM

April 10, 2013 09:00 PM

Analyzing large data sets is hard, but it’s often way harder than it needs to be. Today, Taras Glek and I are unveiling a new data-analysis tool called jydoop. Jydoop is designed to allow engineers without any experience in HBase/Hadoop to write data analyses on their local machine and then deploy the analysis to our production Hadoop cluster. We want to enable every Mozilla engineer to use telemetry and crash-stats data as effectively as possible.

Goals

Jydoop started with three simple goals:

Enable fast prototyping/testing

Setting up hadoop/hbase is unreasonably difficult and time-consuming, and engineers should not need a hadoop/hbase setup in order to write an analysis. Because Jydoop analyses are written in Python, they can be developed and tested on a local machine without Hadoop or even Java.

Don’t hide map/reduce

In a query language like pig, it is hard to know exactly how your query will perform; which tasks will run on the map nodes, which will run on the reducers, and which must be run on the final reduced data. Jydoop doesn’t hide those details: each analysis has simple map/combine/reduce functions so that engineers know how a clustered query is going to behave.

Be fast

Performance of clustered queries should be as good or better than our existing solutions using pig or other existing tools.

Prototyping

Telemetry data takes the form of a single blob of JSON which is stored in an hbase table. A telemetry ping may be larger than 100kb, and we typically receive about 2 million telemetry pings per day. In order to allow engineers to test an analysis, we save off a small sample of reports (5000 or so) in a single file. Then analysis scripts can be written and tested against the sample.

The simplest job is one which simply acts as a filter. The following script is used to select all telemetry records which have an “androidANR” key and save them to a file:

import telemetryutils

# Ask telemetryutils to set up our query by date range using the correct hbase tables
setupjob = telemetryutils.setupjob

def map(key, value, context):
    if value.find("androidANR") == -1:
         return

    context.write(key, value)

To test this script against sample data, run python FileDriver.py scripts/anr.py sampledata.txt.
To run this script against a single day of telemetry data, run make ARGS='scripts/anr.py anrresults-20130308.txt 20130308 20130308' hadoop.

More complex analyses are also possible. The following script will calculate the bucketed distribution of the size of the telemetry ping:

import telemetryutils
import jydoop
import json

setupjob = telemetryutils.setupjob

kBucketSize = float(0x400)

def map(key, value, context):
    j = json.loads(value)
    channel = j['info'].get('appUpdateChannel', None)

    # if .info.appUpdateChannel is not present, we don't care about this record
    if channel is None:
        return # Don't care about this record

    bucket = int(round(len(value) / kBucketSize))
    context.write((channel, bucket), 1)

combine = jydoop.sumreducer
reduce = jydoop.sumreducer

I was then able to produce this chart with the result data and a little JS:

History and Alternatives

The native language for Hadoop map/reduce jobs is Java. Writing an analysis directly in Java requires careful construction of a class hierarchy and is extremely tedious. Even more annoying, it’s basically impossible to test an analysis without having access to a hadoop/hbase environment. Even if you use something like the cloudera test VMs, setting up hbase and mapreduce jobs on test data is onerous.

It is a venerable Hadoop tradition to build wrapper tools around mapreduce which run distributed queries. There are tools such as Hadoop Streaming which allow map/reduce to forward to arbitrary programs via pipes. There are at least five different tools which already combine Hadoop with python. Unfortunately, none of these tools met the performance and prototyping requirements for Mozilla’s needs.

At Mozilla, we currently mostly use pig, a query language and execution tool which transforms a series of query and filter statements into one or more map/reduce jobs. In theory, pig has an execution mode which can prototype a query on local data. In practice, this is also very difficult to use, and so most people prototype their pig scripts directly on the production cluster. It works after a while, but it’s not especially friendly.

Performance

They key difference between Jydoop and the existing Python alternatives is that we use Jython to run the python code directly within the map and reduce jobs, rather than shelling out and communicating via pipes. This allows for much tighter integration with hadoop and also allows us to control the execution environment.

For maximum performance, we use a custom key/value class which serializes and compares python objects. For performance and sanity, this class operates on a limited subset of Python types: analysis scripts are limited to using only None, integers, floats, strings, and tuples of these basic types as their key and values.

During the process of developing Jydoop, we also discovered that JSON parsing performance was a significant factor in the overall job speed. The performance of the `json` module which is built into jython 2.7 is horrible. We got better but not great performance using jyson as a drop-in JSON replacement. Eventually we wrote a complete json replacement which wraps the standard Jackson streaming API. With these improvements in place, tt appears that jydoop performance beats pig performance on equivalent tasks.

Conclusion

Jydoop is now available on github.

We hope that jydoop will make it possible for many more people to work with telemetry, crash, and healthreport data. If you have any issues or questions about using jydoop with Mozilla data, please feel free to ask questions in mozilla.dev.platform. If you are using jydoop for other projects, feel free to file github issues, submit github PRs, or email me directly and let me know what you’re up to!

April 09, 2013 05:52 PM

As you might have gathered, I am getting ready to land a major set of changes. This set of changes is being tracked in bugs 790855, 842632, and 858337. These patches are implementing structured header parsing and emission, as well as RFC 2047 decoding and encoding. My goal still remains to land all of these changes by Thunderbird 24, reviewers permitting.

The first part of JSMime landed back in Thunderbird 21, so anyone using the betas is already using part of it. One of the small auxiliary interfaces (nsIMimeHeaders) was switched over to the JS implementation instead of libmime's implementation, as well as the ad-hoc ones used in our test suites. The currently pending changes would use JSMime for the other auxiliary interfaces, nsIMimeConverter (which does RFC 2047 processing) and nsIMsgHeaderParser (which does structured processing of the addressing headers). The changes to the latter are very much API-breaking, requiring me to audit and fix every single callsite in all of comm-central. On the plus side, thanks to my changes, I know I will incidentally be fixing several bugs such as quoting issues in the compose windows, a valgrind error in nsSmtpProtocol.cpp, or the space-in-2047-encoded-words issue.

It's not all the changes, although being able to outright remove 2000 lines of libmime is certainly a welcome change. The brunt of libmime remains the code that is related to the processing of email body parts into the final email display method, which is the next target of my patches and which I originally intended to fix before I got sidetracked. Getting sidetracked isn't altogether a bad thing, since, for the first time, it lets me identify things that can be done in parallel with this work.

A useful change I've identified that is even more invasive than everything else to date would be to alter our view of how message headers work. Right now, we tend to retrieve headers (from, say, nsIMsgDBHdr) as strings, where the consumer will use a standard API to reparse them before acting on their contents. A saner solution is to move the structured parsing into the retrieval APIs, by making an msgIStructuredHeaders interface, retrievable from nsIMsgDBHdr and nsIMsgCompFields from which you can manipulate headers in their structured form instead of their string from. It's even more useful on nsIMsgCompFields, where keeping things in structured form as long as possible is desirable (I particularly want to kill nsIMsgCompFields.splitRecipients as an API).

Another useful change is that our underlying parsing code can properly handle groups, which means we can start using groups to handle mailing lists in our code instead of…the mess we have now. The current implementation sticks mailing lists as individual addresses to be expanded by code in the middle of the compose sequence, which is fragile and suboptimal.

The last useful independent change I can think of is rewriting the addressing widget in the compose frontend to store things internally in a structured form instead of the MIME header kludge it currently uses; this kind of work could also be shared with the similar annoying mailing list editing UI.

As for myself, I will be working on the body part conversion process. I haven't yet finalized the API that extensions will get to use here, as I need to do a lot of playing around with the current implementation to see how flexible it is. The last two entry points into libmime, the stream converter and Gloda, will first be controlled by preference, so that I can land partially-working versions before I land everything that is necessary. My goal is to land a functionality-complete implementation by Thunderbird 31 (i.e., the next ESR branch after 24), so that I can remove the old implementation in Thunderbird 32, but that timescale may still be too aggressive.

April 07, 2013 09:47 PM

Firefox on desktop has nice support for previewing and subscribing to syndication feeds (RSS and Atom). There is even support for creating Live Bookmarks. Firefox for Android does not support anything related to syndication feeds… until now.

We just landed basic support for subscribing to feeds discovered on a web page. This is only initial support, so many things are not supported.

How it works:

• If feeds are discovered on a page, Firefox will enable a menu action on the URLBar long-tap menu. Yes, we have a URLBar long-tap menu!
• Tap the “Subscribe to Page” menu action.
• If there is more than one advertised feed, choose the feed you want.
• Pick the online web service where you want to add the subscription. The choices really depend on the locale, but initially Google Reader (but not for long) and Yahoo are supported.

The limitations:

• We only support online web services right now. We are looking into local application support too, but have not seen a universal way of listing feed reader applications. We are working on a proposal and want to get support from native feed reader apps like Feedly and Newsblur.
• No support for feed previews. No plan to add support
• No support for adding new web service handlers. We are working to add this support.
• No support for Live Bookmarks. No plan to add support.

In pictures:

Long-tap on the URLBar

Choose “Subscribe to Page”

Choose the feed

Choose the web service

April 04, 2013 06:56 PM

On The Web

If you want to stay up to date on any new developments in Firefox for Android (Fennec), check out the new Tumblr and Twitter stream. Lots of updates on what’s landing in Fennec Nightly, tips and tricks and summaries of the Mobile Engineering Team meetings.

Mailing List

We have a new mailing list at mobile-dev-firefox@mozilla.org. The newsgroup at mozilla.dev.platforms.mobile is being closed. Use the new mailing list to following along and give feedback on topics being discussed, or post your own ideas.

On IRC

As always, you can jump on Mozilla IRC and talk directly to the Mobile team in the #mobile channel. This is great if you want to start working on the code, need help writing an add-on, want to help investigate a bug or just rant about some design decision.

April 04, 2013 01:17 PM

I’ve been thinking quite a bit lately about how Webmaker can take advantage of the work being done on Web Components and Custom Elements.  I tasked Pomax with doing some research and prototyping, and the results have been encouraging.  I wanted to say something about my current thinking and what we might do.

One of the successes we had with Popcorn.js was that it became really easy to create plugins that could take care of the complexities of working with third-party APIs and showing web content dynamically, and linearly in time based on media resources.  Consider the Google Map plugin, which allows one to specify a location, map type, etc. and have a map dynamically load within the provided container div:

var p = Popcorn( "#video" )
  .googlemap({
       start: 5,
       end: 15,
       type: "ROADMAP",
       target: "map",
       lat: 43.665429,
       lng: -79.403323
  });

We’ve seen people make all kinds of plugins to simplify the use of more complex APIs like this.  Often it means dynamically loading some script, setting up a div or iframe, adding an element with a particular class name or id, and applying some styles.

There are a few things I’ve always wanted to improve about it, though.  First, how to properly combine a bunch of styles with a plugin.  Currently a plugin like the googlemap one is just JavaScript.  In reality, you almost always want a mix of JS and CSS, and doing CSS in script is not ideal; nor is having to manage the loading of two files per plugin (*.js, *.css).  A second issue is that these plugins are tightly bound to Popcorn, which is fine if you’re time-based experiences, but not so good for a normal web page.

With custom elements, we can solve a lot of these problems.  First, a custom element is a simple HTML file containing the definition of a new element, any associated styles, and scripts.  I use it in my page by loading it via a link:

<link rel="component" href="webmaker-components.html">

I love the simplicity of this method of packaging and loading new elements.  This file can include one ore more new elements, making it possible to do some server-side bundling for dynamic builds–imagine loading components a, b, c, and d via http://components.webmaker.org?c=a,b,c,d.

Now in your markup it’s possible to take our Google map from above, and rewrite it like so:

<webmaker-map start="5" end="15" location="Maruyama Zoo">
</webmaker-map>

Here content attributes on the custom webmaker-map element provide a way to indicate start and end times, and a location to pass thorugh to Google’s API.  However, while I can specify timing info, I can also leave it off, indicating that the map should always be visible:

<webmaker-map location="Maruyama Zoo"></webmaker-map>

By moving the functionality of the Google Map plugin to a custom element, we’ve made it possible for it to be used in any web context or tool.

You can see this in action in Pomax’s demo page: first with timing info, second without as a static web page.  View the source of both pages.  First off, notice how little code there is, and more, how it’s all markup vs. script.  You might be wondering why this works at all, since browser vendors are still defining and implementing the spec.  You’ll see that this uses a polyfill to give us the ability to play with things now.

This model of providing functional-units to authors is also exciting in the way it lets us mix and match depending on the needs of the individual.  For example, lets say you want to add comments to your page.  One option would be to use Facebook’s comments plugin, another would be to use Disqus’s embed.  Both can be wrapped into custom elements, and users can choose.  I did just that for Facebook’s comments: code (view source) and demo (scroll to the bottom).  Here I use

<facebook-comments></facebook-comments>

to add the functionality I want.  No scripting of any kind, which means that this becomes accessible to so many more web makers, for whom coding is beyond their reach.  Whenever we can reduce complexity like this, we should do it.

I think moving toward HTML-based custom elements vs. script-only plugins is going to make a huge difference.  It’s going to mean that it’s easy to integrate with existing tools and workflows, easy to find and use them (they are just web pages, after all), and most of all, easier for non-devs to use all these amazing APIs and services, which require coding.

Pomax and Matt have already started work to convert Popcorn Maker to use this, and are making good progress.  I’m looking forward to building more pieces of the Webmaker tools and gallery infrastructure using these same methods, and I’ll write more about or progress later when I have something to show.

April 02, 2013 08:01 PM

I wrote previously about our work to add Media Clips to Popcorn Maker. Since then we’ve watched as people have started to make things with it, from Ron Swanson dancing to any song to the International Space Station orbiting Earth, which has been a lot of fun.

One project Brett showed me today was done by BBG as a procedural storytelling experiment using a SoundCloud audio monologue about the election of the new Pope.  It’s a really good example of how one can take a bunch of source media (multiple YouTube clips, images, text, audio clips) and mix them together into a more complete, seamless, web-media experience.  What’s even better is that you can click “Remix” and see how it was done.  I’ve done just that in the image below, which shows the media clips they’ve used in the Media Gallery, and the timeline where they get used.

I love the final effect–it really feels like something new, something I don’t see on the web right now: people have all kinds of tools for telling stories spatially, but not in time, linearly.  We’ve long had people making fully rendered videos, but that’s not what this is: this is taking media and resources from across the web and mixing them together in the browser.  We’re still testing and evolving this, but already it’s getting to be quite usable.

I’m equally interested to see what people will do with another experiment we’re trying, using YouTube’s webcam uploader.  I think when we combine user created media with the ability to mix audio and video from other sources, we’re going to have something really interesting.  Almost there…

March 27, 2013 07:38 PM

Short version: I’m hiring, apply here!

Do you love your debugger? Are you interested in a job making a better Firefox for millions of people? Making Firefox a rock-solid software program involves being a dedicated puzzle-solver. Figuring out what’s causing a crash, hang, or intermittent performance problem is a combination of data gathering, statistics, debugging, code reading, and proactive coding and debugging features.

Familiarity or even love of debugging is an important part of the job. The set of information in a crash minidump is limited, and frequently the stack trace is garbled. Understanding calling conventions and memory layout of compiled code is an important skill! Being able to read disassembly is essential. Familiarity with debugging across multiple platforms (Windows/Mac/Linux/Android) is going to be important.

The Mozilla crash and telemetry data is a treasure trove of information just waiting for analysis. The servers have some important builtin features for classification and querying, but the stability team frequently needs to slice the data in new ways. Familiarity with data processing using python, SQL, hbase, and elasticsearch is helpful.

Many of our stability issues are caused by third-party software (more than half of our crashes, according to recent statistics). This position will undoubtedly involve working with partners and engineers from other projects and companies to diagnose and fix problems.

Finally, we are always looking for ways to solve entire classes of stability issues. For example, we moved browser plugins into their own process so that even if a plugin crashes, it doesn’t affect the entire browser. We are continuing to reduce the impact of plugin and extension issues by having users opt into these addons. To be effective at these types of projects, an engineer will need to be a productive coder who is comfortable learning new pieces of code quickly, as well as writing in C++ as well as JavaScript.

If you’re interested, apply online, and feel free to contact me directly if you’ve got questions!

March 07, 2013 06:26 PM

HTML5 includes some great additions for authors wanting to work with microformats and microdata in markup.  Last week I finished implementing the new <time> and <data> elements in Firefox (see bugs 629801 and 839371).  You can already use them in Nightly, and they should ship as part of Firefox 22.  I wanted to say something about what they are, how you can use them, and why I implemented them.

The idea behind both elements is to make it easier for authors to include various kinds of data in markup.  The <data> element enables one to store extra machine-readable data along with textual content.  For example, imagine I want to display a user’s name, but also include their user ID without displaying it:

<data id="user" value="humphd">David Humphrey</data>

The <data> element adds a new attribute, value, which contains a string representation of your data.  In script I can use the .value property to get the reflected value:

var user = document.getElementById("user"),
id = user.value;

In the past people used data-* attributes to do something similar, for example:

<span id="user" data-username="humphd">David Humphrey</span>

This works well, but it introduces a non-standard way of naming such data.  With the <data> element, all such data is accessible via the value attribute and property.

The value of a <data> element can be anything, from part numbers to IDs to co-ordinates–your scripts can parse it out and decide what to do with it. When the value you wish to store is really some form of date or time, the more appropriate choice is <time>.

The <time> element a specialized form of <data> used to add machine-readable dates and times to web pages.  There are a few ways to use it:

<time>2013-03-05</time>
<time datetime="2013-03-05">Today</time>

In both cases a <time> element is used to markup a date.  The latter uses the datetime attribute to record a machine-readable date/time value.  In JavaScript you’d access this value via the element’s .dateTime property (note the use of a capital ‘T’ on ‘Time’).  The .dateTime property reflects the datetime attribute.

I wanted to implement these for a few reasons.  First, one of my classes is implementing the HTML5 <track> element right now, and I wanted to show them how Mozilla’s new WebIDL bindings work with some simple examples.  Second, I wanted to do some work on the ideas Atul mentioned in his blog post about Markup APIs.

One of the goals Brett and I have this year for the Webmaker tools is to have them publish pure markup vs. using JSON or JavaScript.  We want to make it possible to create dynamic web experiences using only HTML and CSS.  I’ll write more soon about what we imagine, but the idea of being able to include start and end times in markup was very interesting to me, since much of our work is building time-based web pages using media with Popcorn.js and Popcorn Maker.

It’s neat to see that people are already using these elements in the wild.  Github, for example, uses <time> to record detailed info about commits, while not cluttering the UI with overly verbose data, for example:

<time class="js-relative-date updated"
datetime="2013-03-05T07:21:28-08:00"
title="2013-03-05 07:21:28">March 05, 2013</time>

I’m looking forward to seeing how people will use <data> as well–I haven’t been able to track down any users of it yet, but I’m sure they’re out there.  If you know of any, throw links in the comments.

Go forth and microformat!

March 05, 2013 08:38 PM

Now, many people have attempted to argue against this risk by one of three arguments: that Webkit already is a monoculture on mobile browsing; that Webkit is open-source, so it can't be a "bad" monoculture; or that Webkit won't stagnant, so it can't be a "bad" monoculture. The first argument is rather specious, since it presumes that once a monoculture exists it is pointless to try to end it—walk through history, it's easier to cite examples that were broken than ones that weren't. The other two arguments are more dangerous, though, because they presume that a monoculture is bad only because of who is in charge of it, not because it is a monoculture.

The real reason why monocultures are bad are not because the people in control do bad things with it. It's because their implementations—particularly including their bugs—becomes the standards instead of the specifications themselves. And to be able to try to crack into that market, you have to be able to reverse engineer bugs. Reverse engineering bugs, even in open-source code, is far from trivial. Perhaps it's clearer to look at the problems of monoculture by case study.

In the web world, the most well-known monoculture is that of IE 6, which persisted as the sole major browser for about 4 years. One long-lasting ramification of IE is the necessity of all layout engines to support a document.all construct while pretending that they do not actually support it. This is a clear negative feature of monocultures: new things that get implemented become mandatory specifications, independent of the actual quality of their implementation. Now, some fanboys might proclaim that everything Microsoft does is inherently evil and that this is a bad example, but I will point out later known bad-behaviors of Webkit later.

What about open source monocultures? Perhaps the best example here is GCC, which was effectively the only important C compiler for Linux until about two years ago, when clang become self-hosting. This is probably the closest example I have to a mooted Webkit monoculture: a program that no one wants to write from scratch and that is maintained by necessity by a diverse group of contributors. So surely there are no aftereffects from compatibility problems for Clang, right? Well, to be able to compile code on Linux, Clang has to pretty much mimic GCC, down to command-line compatibility and implementing (almost) all of GCC's extensions to C. This also implies that you have to match various compiler intrinsics (such as those for atomic operations) exactly as GCC does: when GCC first implemented proper atomic operations for C++11, Clang was forced to change its syntax for intrinsic atomic operations to match as well.

The problem of implementations becoming the de facto standard becomes brutally clear when a radically different implementation is necessary and backwards compatibility cannot be sacrificed. IE 6's hasLayout bug is a good example here: Microsoft thought it easier to bundle an old version of the layout engine in their newest web browser to support old-compatibility webpages than to try to adaptively support it. It is much easier to justify sacking backwards compatibility in a vibrant polyculture: if a website works in only one layout engine when there are four major ones, then it is a good sign that the website is broken and needs to be fixed.

All of these may seem academic, theoretical objections, but I will point out that Webkit has already shown troubling signs that do not portend to it being a "good" monoculture. The decision to never retire old Webkit prefixes is nothing short of an arrogant practice, and clearly shows that backwards compatibility (even for nominally non-production features) will be difficult to sacrifice. The feature that became CSS gradients underwent cosmetic changes that made things easier for authors that wouldn't have happened in a monoculture layout engine world. Chrome explicitly went against the CSS specification (although they are trying to change it) in one feature with the rationale that is necessary for better scrolling performance in Webkit's architecture—which neither Gecko nor Presto seem to require. So a Webkit monoculture is not a good thing.

February 16, 2013 04:39 AM

February 15, 2013 06:37 PM

‘Making is learning’ is a big theme for Mozilla this year. It’s at the heart of Mozilla Webmaker. More importantly, it’s the north star idea guiding the grassroots mentor community we’re building around the world. We want millions more people to get their hands dirty with the web. And we expect they’ll learn something as they do.

I realized today that we need to add two concepts into this theme: tinkering and social. This thought came from a good discussion on the Webmaker mailing list that starts with the question ‘is making learning?’ Rafi Santo both asked and began to answer this question:

The short answer: yes, but it’s complicated. The longer answer is that the best maker-driven learning is never just about the making. It’s about all the things that happen around the making. That initial spark of curiosity, the investigation and early tinkering, the planning and research that follow, the inspirations and appropriations from other projects, the prototypes, the failures, the feedback, and, perhaps most importantly, the iterations upon iterations towards a better make.

He then went on to say:

I’m willing to say that someone is always learning something when they’re making, but they learn best when it entails the sort of process, community and well configured structures of participation.

In part, the discussion around Rafi’s post is a debate about tag lines. Should we rally people under a ‘making is learning’ banner? Or should we be more subtle like ‘making as learning’ or ‘make to learn’? We’ll probably do the later.

However, there are also two important substantive points worth pulling out from the conversation: a) it’s the process of making that drives learning and b) the best learning happens when the making is social. Both of these points are critical to the success of Webmaker.

The process point may be obvious. It’s not just what I made, it’s the journey of the making. But it’s worth calling it out explicitly. Mozilla Rep Emma Irwin writes in response to Rafi’s post:

This spoke to my own learning in programming. I think I learned (and got confidence) more from debugging and being stuck than simply making. The sense of accomplishment of overcoming things that seemed really hard at first have motivated me more than anything. I think those experiences are why I am crazy enough to think I can ‘teach’ now.

Designing tinkering and iteration into Webmaker is critical. A first step is creating content built from the ground up for remix. And, then to support that with tools that let you tinker and play with that content, and share it again with your friends. The idea is to use remix as an onramp to tinkering with the web.

You see an early example in Jacob’s awesome Valentine’s video project on Webmaker.org. The thing about this video: it is designed to be forked. It wants you to add your own photos and change the text. It’s an invitation to tinker. It’s an early invitation, to be sure: we clearly have a lot to learn about how to do this well. But it’s clear to me that this kind of design for tinkering is ‘thing #1′ of key things Webmaker needs to pull in from this conversation.

Rafi’s other big point is about social: we learn best when we make together. Making together can mean a lot of things. At events. In school. With friends at home. In IRC. On Facebook. Etc. What all of these things have in common is that I can see what you are making and you can see me. We can critique each other. We can help each other. We can fail together. We can iterate together. And we can laugh together. Which makes learning funner, faster and deeper.

Making it easy to ‘make things together’ is ‘thing #2′ that Webmaker should pull from this conversation. Making it easy to riff on content on Webmaker.org and in places like Facebook will be a part of this. But, as Rafi hints in his post, the most important factor here won’t be tools and web sites: it will be people. This is why the building a global mentor community is such a huge priority. Everyone needs a place where they can just show up to make and learn. A place filled with people. And a place you can find in 100s of cities around the world. Building on Hive and ReMo, I think Mozilla can create this place. It’s what we want our mentor community to be.

Anyways: thanks Rafi, Emma and others for getting this conversation started. It’s the kind of leadership this nascent Webmaker community needs. And it’s a great way to dig into what do we really want to build together with Webmaker.

February 13, 2013 06:24 PM

One of the most requested new features we get for Mozilla Webmaker is web media sequencing.  Users want to be able to use multiple video or audio clips in a single experience.  This past week Scott and Kate, with help from Matt and Chris, have been building and testing a solution in Popcorn Maker.

The idea is to make it easy to pick media clips, including HTML5 audio and video, YouTube, SoundCloud, and Vimeo, and let users position each clip along the timeline, and within the video canvas.

Here two YouTube clips have been added to the gallery.  I might want to use the video from one (surf video) and the audio from the other (“Wipe Out”).  The sequencer supports video+audio, video-only, and audio-only for each clip.  Next you figure out where to place clips in the timeline:

Here a Vimeo video and an HTML5 video are positioned within the timeline so that the former will start part way through the latter.  We can also specify the position and size of each visual clip within the video canvas, which means we can do interesting parallel layouts of multi-video experiences.:

In this example, four separate YouTube videos are positioned into a common space, with one of them occupying the majority of the space, and thereby emphasizing its importance  Each clip can be resized or moved (you can see that one of the clips is selected in the editor and has resize handles).

The Sequencer feature is still in active development, but is ready to be tested.  You can already make some interesting things with it.  Here are some examples you can look at:

• Clay Shirky talk with Flickr slide show + YouTube b-roll
• Near Death in Bb, mixing multi-track audio video
• Baby Awake vs. Harlem Shake, remixing audio and video

You can remix any of the above, or make your own using this experimental branch.  You can let us know about issues you find or make suggestions in this bug.

February 12, 2013 05:18 PM

I've released a new version of the ntfsutils library for Python.

The small spin-off project from some work I did at Mozilla became far more popular than I expected, netting over 600 downloads last year. Pretty neat!

January 26, 2013 06:30 AM

A while ago, I noticed that my Firefox was starting to run slower and slower. One of the common suggestions to fix that kind of problem is to reset your profile, and while I think that’s a good idea the biggest problem for me was that resetting my profile would not save my open tabs. Since I have 75 of them, trying to remember them all would be a bit of a problem. (Some people would suggest that having 75 open tabs is also a problem, but fie to them I say! Sure, I’m a little bit above the average, but I’ve hit a flow that works for me. :)

Fortunately, I’m a coder at heart, so I did what any coder would do, and wrote some code to save and restore them for me. A few days later, one of my co-workers wanted to do the same thing, so I ran similar code on his computer, and since at least the two of us found it useful, I thought it might be a good idea to post the code here so that other people might benefit from it, too.

First of all, we need a little bit of setup to prepare your computer to save the tabs.

1. Go to Tools » Web Developer » Scratchpad to open up the Scratchpad. (This is where we’ll be doing most of our work.)

2. Click Environment » Browser to make sure the Scratchpad is running in the browser’s context, so that it can access the list of tabs.

3. Paste the following code into the Scratchpad:

   var x = gBrowser.tabs;
   var rv = 'var tabs = [\n';
   for (var i = 0; i < x.length; i++) {
     rv += '  "' + x[i].linkedBrowser.contentWindow.location + '",\n';
   }
   rv += '];\n';
   rv += 'for (var i = 0; i < tabs.length; i++ ) {\n';
   rv += ' gBrowser.addTab(tabs[i]);\n';
   rv += '}\n\n';
   
   rv
   

4. Click Execute » Display This will output stuff in comments (“/*…*/”) that looks like this:

   var tabs = [
     "http://twitter.com/bwinton",
     /* … */
     "http://weblog.latte.ca/blake/tech/",
   ];
   for (var i = 0; i < tabs.length; i++ ) {
    gBrowser.addTab(tabs[i]);
   }
   

5. (This is the big step!) Save that code to an external file, and run the profile reset.

6. Then, in your new profile, open the Scratchpad, and set the Environment to Browser again just like above, and paste in the code you saved.

7. Finally, hit Execute » Run, and all your tabs should be restored! (If you’ve split them up into tab groups, the code I provided will save all your tabs, but will restore them all into one group, so you’ll need to re-group them after restoring. I thought about fixing the code to save a list of groups, but I didn’t really need to, so never got around to it. ;)

I hope this helps some of you, and please feel free to leave a comment letting me know whether you found it useful or not, or if you come up with any additions or changes to make it work better!

January 23, 2013 05:18 PM

The #ifdef problem

In discussing DXR at the 2011 LLVM Developers' Conference, perhaps the most common question I had was asking what it did about the #ifdef problem: how does it handle code present in the source files but excluded via conditional compilation constructs? The answer then, as it is now, was "nothing:" at present, it pretends that code not compiled doesn't really exist beyond some weak attempts to lex it for the purposes of syntax highlighting. One item that has been very low priority for several years was an idea to fix this issue by essentially building the code in all of its variations and merging the resulting database to produce a more complete picture of the code. I don't think it's a hard problem at all, but rather just an engineering concern that needs a lot of little details to be worked out, which makes it impractical to implement while the codebase is undergoing flux.

Documentation

Documentation is an intractable unsolved problem that makes me wonder why I bring it up here…oh wait, it's not. Still, from the poor quality of most documentation tools out there when it comes to grokking very large codebases (Doxygen, I'm looking at you), it's a wonder that no one has built a better one. Clang added a feature that lets it associate comments to AST elements, which means that DXR has all the information it needs to be able to build documentation from our in-tree documentation. With complete knowledge of the codebase and a C++ parser that won't get confused by macros, we have all the information we need to be able to make good documentation, and we also have a very good place to list all of this documentation.

Indexing dynamic languages

Here is where things get really hard. A language like Java or C# is very easy to index: every variable is statically typed and named, and fully-qualified names are generally sufficient for global uniqueness. C-based languages lose the last bit, since nothing enforces global uniqueness of type names. C++ templates are effectively another programming language that relies on duck-typing. However, that typing is still static and can probably be solved with some clever naming and UI; dynamic languages like JavaScript or Python make accurately finding the types of variables difficult to impossible.

Assigning static types to dynamic typing is a task I've given some thought to. The advantage in a tool like DXR is that we can afford to be marginally less accurate in typing in trade for precision. An example of such an inaccuracy would be ignoring what happens with JavaScript's eval function. Inaccuracies here could be thought of as inaccuracies resulting from a type-unsafe language (much like any C-based callgraph information is almost necessarily inaccurate due to problems inherent to pointer alias analysis). The actual underlying algorithms for recovering types appear known and documented in academic literature, so I don't think that actually doing this is theoretically hard. On the other hand, those are very famous last words…

January 21, 2013 05:57 PM

... Now toggle the switch! ...

For education, these experiments are simple, quick, and don't require a lot of background knowledge: the student needs only a handful of basic bash commands (cd, ls, cat, echo). Unlike an Arduino, the Pi doesn't need a separate system to host development. You also don't need to deal with files, interpreters, shebang lines, permissions, or compilers. But eventually (and usually pretty quickly), students will want to learn those concepts. In order to save their commands across boots, for example, they will soon want to store them in files: voila, scripts!

It's logical and easy to progress from controlling a single LED and reading a single switch to controlling six LEDs - enough for a two-way traffic light - and then you can add things like pedestrian crossing buttons. Or you can use two infrared LEDs and two infrared phototransistors (which act exactly like switches), mounted in a doorway, to count the number of people that have entered and exited from a room, turning on the lights whenever people are present. These types of projects are fun and engaging ways to teach logic, programming, and circuits.

After a while, students want to do something they can't easily do in bash, like drive a GPIO faster, or poll some complex combination of pins – and they're on to Python (or C, or Perl, or any of a multitude of other languages).

When students/hackers/makers want to connect something more complex than can be easily interfaced through GPIO, the Pi offers serial ports (you can put a message on an LCD display with two bash commands), I²C, and SPI interfaces. And although the ARM processor in the Pi is fairly slow, it is fast enough to do interesting things like speech synthesis and machine vision.

January 18, 2013 08:03 PM

One of the discussions happening right now in the Mozilla Foundation software team is whether mandatory code reviews are a good thing.  I’ve had versions of this conversation a number of times in the past few months, and today I’m going to write my thoughts down so I can point at them when it comes up in the future.  Before I begin, I’ll be honest and voice some frustration, and wonder aloud why this is even necessary.  To me, the question of code review is resolved: you should do it, and the effects on your code, community, and project will be positive.  However, I think it’s a useful exercise to defend things you hold to be true; so here are some of my thoughts on the subject.

My introduction to code review, and the type I know best, is the one practiced by the Mozilla project, specifically what happens with code changes making their way into Gecko, Firefox, etc.  The easiest way for me to describe what happens is to show you some examples.  Before I started writing this, I went and picked a handful of the most recent changesets landed in the mozilla-central repository.  The first thing to note is that these changes explicitly point at the review process, calling out the the reviewer by name:

Along with who wrote the code, the bug being fixed, and a description of the problem, we also see who did the review, r=reviewer.  The person who wrote the code, and the person (or people), who did the review, both show up in the history of the code.  That’s a fundamental part of how we work.

What kinds of code needs a review?  What does it take for that code to get an r+ instead of an r-?  Let’s look at a few real-world cases, starting with Bug 830871.  Here’s a bug about evolving code, laying ground work for things that need to happen down the road.  It’s a small but common type of bug: We used to do X, now we want to start doing Y.  Let’s take a look at the fix:


-extern JS_FRIEND_API(bool)
+extern bool


This fix removes 15 characters. It’s trivial; hardly worth bothering anyone with, so why not just land it without review? Jeff, who wrote the patch, is an expert developer. Jason, who is being asked to spend time reviewing this, is incredibly busy with much more important patches of his own. So what’s going on here?

The first thing to understand about code review in a project with more than one developer (Mozilla has hundreds if not thousands), is that it communicates change. This bug is a perfect example. In the past, some decision has been taken (“We want to get rid of JSProtoKey as part of the JSAPI eventually“) and this is one step on the path to arriving at that change.  We get closer to understanding what’s happening here if we think about code review for such a patch as a kind of Catechism, where an assertion is made publicly, then affirmed.  By reciting what we believe about the direction that the code is taking, we bolster understanding and awareness.  “Folks, remember that we’re changing this API.”

These so-called Catechism reviews can be seen all over the place in Mozilla.  Here’s another one that landed a few minutes earlier.  Again, a tiny change that simply removes a few lines of code, and the desire for confirmation from a trusted peer: “ted: Please rubber stamp trivial change.” Code review spreads awareness of changes across the community.  I’m changing X, and I want you to know about this, too.

Notice also a secondary review request in that bug, this time for testing validation: “gaston: Please verify this actually fixes the problem.“  Here we see the role of review in helping test conditions beyond the means of the developer.  I wrote this patch on platform X, can you test it on platform Y?  Review distributes, shares resources, increases the power and reach of an individual to test their own assumptions.  The review ends with a moment of encouragement, “That allow m-c to pass configure & start building (so fixes the regression from 784841), and i can even use mach now \o/”

Another type of review you’ll find happening a lot is one in which a strategy is developed, an architecture established, or an approach agreed upon.  In the first review I examined above, there were hints of this from the past, “We want to get rid of JSProtoKey…“  At some point that decision was made, and what was being reviewed was more the decision than the code as such.  Code enacts the decision, but the strategy has to get signed-off on as well.  You see a bit of this happening in Bug 831188.  Again, a one line patch (if you’re starting to feel like Mozilla developers never write big patches, don’t be fooled), and this time a review from one of Mozilla’s senior developers and architects.  Benjamin goes on to not only r+ the patch, but also takes the opportunity, through the review, to spec out some new work.  Code reviews are decision points, moments where the direction of not only this code, but that other code we should do, come into view.  What’s more, they are public decisions, which enable participation.

A lot of people decry code reviews as nothing more than busy-work, since many reviews do little more than identify so-called nits.  Having the work of hundreds of developers all adhere to certain standards, and display a common style, etc. means that there will inevitably be things you have to change in order to get your code landed in our code.  I write code for so many different code bases that I don’t know what my coding style is anymore–I do what the current code does.  Nits don’t bother me, and mainly because I know that they often evolved with good reason.  Here’s an example in bug 828228:

One other nit: I don’t like using constants for short strings, like TELEMETRY_SERVICE_INIT/TELEMETRY_BUILD_CACHE – it’s a level of indirection that makes the code harder to read/grep, and doesn’t really serve a purpose (typos would be caught otherwise, and we don’t need to change these values often).

“We’ve found that if you do it this way, it’s better, and here’s why…” Often the rationale for a nit isn’t given (sometimes it’s clearly understood, and just an oversight by the developer). Gavin chooses to use this code review to not only get his way with the code, but also to help educate another developer as to why this is practiced. Code review is where one learns how and why we do certain things.

How about a larger code change? Sure, bug 824864 will do.  Go scan that bug, look at all the various interactions.  What do you see?  First, notice that Bobby confirms the code is passing automated tests.  This review isn’t going to be about “does it work?” since that’s already been established.  Here, code review is about how and why things have been done.  What else do you notice?  I notice Bobby asking for review from two different people.  Next I see three separate reviewers helping out with the effort.  It’s a big and complicated set of changes.  Code review is something we need to do.  What do the reviewers find?  Style issues, re-introduction of unwanted APIs/code, things this change exposes we should follow-up on, changes that maybe shouldn’t be made, brittle code, inconsistent use of APIs, and on it goes.

A reviewer like Boris (truth be told there probably isn’t a “like Boris” anywhere) spends a lot of his time just doing reviews of this kind.  I’ve had people tell me that so-and-so is spending too much time on reviews, implying that if they were writing code instead, they’d be more productive.  The thing this misses is how one developer can only write so many patches, but their influence and ability to affect many more is increased through reviews: Boris doesn’t need to write every line of code, but the 4 or 5 he discusses in a review can make all the difference.  Code review is leverage, which allows top people to be in more than one thing at a time.

It’s important to state that all of the people involved in that last bug are expert developers.  How many times have I heard people claim that their code doesn’t need to be reviewed?  Don’t believe it.  Code review has nothing to do with how good you are as a programmer and everything to do with how complex programming is, and how impossible to get right.  When Brendan (yes that Brendan) writes code, it goes through review too, fails, and gets better.  Code review isn’t something that gets done to you, it happens to the code you write.  It’s not a personal assessment.  It’s something we do in order to not get crushed under the complexity of what we’re building together.  Programming is hard, and Mozilla code reviews are filled with people missing things that others help them see.

Code review is humbling, and it’s good to be humbled.  Code review is educational, and it’s good to be taught.  Code review is public, and it’s good for open source to happen where people can see what’s going on.

I also think that code reviews can be done badly.  People can use code review as a weapon to endlessly tie-up some change from ever happening.  Some might argue that the examples I chose above, while current, are all changes that actually landed.  What about those that didn’t, got WONTFIXED, or the like?  I would argue that the failures there are not about code review, and only show up because of the public nature of a review.  If person A and B are fighting, that review may expose things.  If one camp is against using tool X and others are trying to get it landed, the real issue is communication between those two groups.  Things come to a head in review, which can be uncomfortable, but is also necessary if those issues are latent within the community, project, company.

You can and should call out bad review practices.  Is someone being a jerk with nits?  Talk to them about it.  Is someone infinitely delaying a feature with pointless follow-up review failures?  Go have the real discussion that should be happening instead of the secondary arguments about the code.  The problem isn’t code review, so don’t lay it there.

Code review is also one of the best ways to enable participation, and though there are many other things I could say, I’ll end on this one.  Why not just use pair-programing?  Why not talk to so-and-so at her desk, and get some advice before doing it myself?  Code review, at least in the Mozilla context, happens in public view.  The evolution of the code is something I, as a non-employee, can witness, understand, discuss, and participate in productively.  If we replace code review with higher-bandwidth interactions, which let’s face it means things happening at the office, we eliminate the potential for the same degree of open collaboration.

Mozilla taught me the value of code review a long time ago, and now I want to help continue that culture within the Foundation.  It’s easier to work alone, or in pairs, to not have to pass everything by other people.  It’s also how you stay small and insular.  If you want to grow, if you want to be open, if you want community, and if you want to be Mozilla, I think code review is one of the best ways to get there.

January 18, 2013 06:19 PM

Last month, Anil Dash wrote The Web We Lost. It struck a chord: making the case that there has been t a subtle but massive shift on the web. Not simply a shift from open to closed. It’s more nuanced. Rather, a shift a web that is more human and craftlike to one that is more mechanical and industrial. My words, not his.

As I read it, I was struck by a) how Dash’s post is very much at the heart of why we’re doing Mozilla Webmaker and b) that we’ve done a very poor job ourselves of explaining that why. Which is the reason I’m reaching out to you. I’m working with a bunch of people to explain why anyone should care about web literacy. I need your help.

Myself and others on the Webmaker team came up with this ‘five liner’ to explain the why of what we’re doing.

1. Our goal: help 100Ms more people become makers who understand and tap the full power of the web.

2. Why? The web has fueled massive creativity, productivity and wealth. We want this to continue.

3. When the web was young: people looked under the hood, figured out how it worked and made things.

4. This ‘just figure it out and make it’ is harder to come by today. The hood is harder to open. Learning as you go is not so easy.

5. Mozilla want to turn this upside down. We want to make it easy easy to open again, to learn how things work and to tap the full power of the web.

It’s rough, for sure. But, if you look at the middle three points, you get the idea. The web has given lots to humanity. That happened because it was open and learnable. Now it’s more closed and hard to learn. Which, by implication, puts all the stuff we’ve gotten from the web at risk. Or something like that.

Over the coming weeks, Paula, Ryan, Geoff and I are going to refine the explanation above and add meat below each bullet. As we do this, we’re looking for help on three questions:

a. For the bullets above, what evidence or examples would you add to bring these points to life? E.g. the private sector has seen 13% rise in productivity due to the web.

b. What other top level arguments would you make for massive web literacy? Why should people care about 100s millions more people understanding how the web works?

c. Most important: how do we deal with the YouTube/Facebook factor? I.e. people are already ‘making stuff’ on social media. What are we talking about that is different? What does ‘the full power of the web’ really offer?

Cracking this nut is important to us. The ideas of ‘web literacy’ and ‘making as learning’ are going to be part of big campaigns we and others do this year. Succinctly explaining why these things matter is going to be critical to success.

Any ideas or comments you’ve got, not matter how hair brained they seem, are helpful. Please leave comments below. Or add to this etherpad.

January 16, 2013 09:55 PM

Matt Thompson just posted the Mozilla Webmaker 2013 plan slides from our last board meeting. As he summarizes:

Webmaker teaches the art 
and craft of webmaking to anyone who wants to make something on the web. It starts with projects: users make (amazing) things, 
learning about the technology and culture of the web as they go.

In 2012 we built the Webmaker brand, product and community.
In 2013 we’ll refine, recruit and get more people using it.

Big picture goal for 2013: Turn our basic Webmaker offering into 
a product people love. Refine it, recruit mentors and get more people 
using it.

These slides represent the culmination of the Webmaker 2013 planning work we started late last year. More importantly, they are the foundation for the next generation products and community we’re building. There are already a number of good blog posts about what we’re building next here, here and here.

January 16, 2013 09:30 PM

Today, and as I write this, friends and family of Aaron Swartz are gathering to pay their last respects to a son, friend, and colleague.  Our Mozilla Foundation software team call, which normally happens at this time, was rescheduled in order to allow a number of the engineers to attend the service in Chicago, and for others to mourn and reflect privately.

I have been reading and pondering the many tributes, interviews, and articles about Aaron, his case, and death.  I didn’t know him personally, but I am familiar with his work and place within our community.  I am also deeply aware that Aaron was an archetypal figure, the 26 year old genius hacker–I’ve taught and worked with literally hundreds and hundreds of people like him in the open source world.  He has come to represent our community, which is, I think, why his death has affected so many, including myself.

The post that has stuck with me, and the reason why I write today, was Lessig’s.  In it, he writes:

Aaron had literally done nothing in his life “to make money.” He was fortunate Reddit turned out as it did, but from his work building the RSS standard, to his work architecting Creative Commons, to his work liberating public records, to his work building a free public library, to his work supporting Change Congress/FixCongressFirst/Rootstrikers, and then Demand Progress, Aaron was always and only working for (at least his conception of) the public good. He was brilliant, and funny. A kid genius. A soul, a conscience, the source of a question I have asked myself a million times: What would Aaron think? That person is gone today, driven to the edge by what a decent society would only call bullying. I get wrong. But I also get proportionality. And if you don’t get both, you don’t deserve to have the power of the United States government behind you.

Proportionality–that word has not left my mind since first reading this post on Saturday.  I’ve since read the outcry and petitions against the government prosecutor.  I don’t know the law, and won’t comment on a case I can’t really understand.

And yet I am moved by Lessig’s call, first for Aaron, but applicable far beyond his case, to practice proportionality, restraint, good judgement, and to examine the power one has, to consider its effects.  I could examine and criticize the power of this prosecutor, but I am much more able to examine the powers granted to me in my own life as a father, professor, project leader, etc.  The call for proportionality is an important one, and the lesson laid bare by Aaron’s case and subsequent death, is one I can and should apply to myself.

We need to be aware of the effects of our actions, and how our will goes out into the world.  Whether it be in grading, discipline, reviewing, judging–we are not free from the impact of our actions, and are never wholly in control of their outcomes.  We need, I need, to always be aware of the humanity at the receiving end of  power.

I pray that I would be slow to exercise power, and that love and forgiveness guide my hand.

Rest in peace, Aaron.

January 15, 2013 04:23 PM

I was reading scrollback on irc this morning, where a number of my students were discussing the unit tests for the webvtt project we’re doing this term.  The main issue was that Visual Studio didn’t seem to provide the equivalent of our autotools build system’s `make check‘, that is, it can’t run all our tests and report back.  Various solutions were discussed, some sensible, some less so.

Our autotools build actually works in Windows; you can `make check‘ to your heart’s content.  The real issue here is, in my opinion, an unhelpful (I would go so far as to say ‘harmful’) reliance on Visual Studio.  I make this bold claim because I was once in the same position.  I can remember very well being wholly, and completely dependent on Visual Studio for just about everything I did as a programmer.  And why not?  Visual Studio is an amazing tool, right?

The longer I’ve programmed, the more I’ve come to see it as a trap.  Not because it costs money, or because Microsoft makes it, or because it isn’t open source.  I’m writing this blog post on a Mac, so I’ve already made my peace with pragmatism over Free as in Open.  No, the trap of Visual Studio is being able to do only what Visual Studio can do, which seems like a lot, until you need to start doing things it can’t do, or worse, things it doesn’t want you to do.

I spend much of my time training new developers, and what I’m going to say will seem to run counter to the teaching of my colleagues.  I want to be clear that I think you should learn how to use Visual Studio, if for no other reason than to learn how its debugger works.  While you’re learning and using that debugger, understand what’s happening; understand that as a programmer you have a set of needs, and debugger is one of the main ones.  Now spend some time with Visual Studio’s editor.  It’s pretty good.  While you’re doing this, understand that you’re using an editor, that this is another of the fundamental needs you’ll have for your entire career.

Take an approach to using Visual Studio that makes you aware of its constituent parts, and less as a monolithic whole.  Why should you do this?  It’s important to understand that you need various kinds of development tools, from editors to debuggers to compilers to linkers (quick, what is the name of the compiler and linker on Windows?) to static analysis tools of all kinds, and that Visual Studio provides some of these.  It does not provide them all.  Not by a long shot.

When this isn’t done, and Visual Studio is viewed as a whole, as a development platform, as the way I write code, a number of immediate consequences follow.  First, the only kinds of programs you’ll write are those which Visual Studio can produce.  In essence, your platform isn’t even Windows any more, it is Visual Studio.  Want to work on OS X?  What about Linux?  What about embedded systems?  What about gaming consoles?  The scope and scale of your work as a developer becomes the answer to the question, “Does it run Visual Studio?”

If, on the other hand, you start to understand Visual Studio as a set of tools, and more, a set of replaceable tools instead of your go-to-dev-platform, interesting possibilities arise.  You might notice, for example, that you’re writing code in C or C++, and this code is being compiled by cl.exe.  What if you switched compilers?  What if you used clang or gcc?  What if you tried a different editor, one that also worked on other platforms, one that supported languages Visual Studio doesn’t seem to know about?

A number of things happen in that moment.  Suddenly, and it is like opening a dam, a flood of new tools, languages, and platforms comes into scope.  Maybe you’ve avoided trying some new language or tool that others are raving about, putting it off because it didn’t seem to fit into the world-view that Visual Studio provided.  Don’t tell me you haven’t because I know it to be true–I used to do this all the time.  If your choice of tooling, platform, language, etc. begins with a question about Visual Studio, you are missing out already.

And what am I advocating instead of Visual Studio?  Should you do what I do?  Probably not.  I use the command line and emacs for everything, and there are probably better options for someone starting today.  Whatever you pick, be aware of the limitations your choices impose on you.  I still use emacs because it’s important to me to be able to SSH into remote boxes and work identically to how I work locally.  I also choose to only work with tools that work on Windows, Mac, and Linux, so I can be productive everywhere.  If I was starting today, I’d probably use Sublime.

It doesn’t matter what you use, and the best programmers I know change their tools and platforms all the time, making decisions based on project needs vs. what their old tools can do.

You should use what works until it doesn’t, and then move on to other things.  Don’t be nostalgic.  Your life as a programmer will involve constant changes in languages, tools, and platforms, and if the only thing making that decision for you is Visual Studio, you will quickly lose the agility you need to stay relevant in the industry.

January 09, 2013 04:01 PM

One of the things I’m currently doing is helping the Mozilla Foundation figure out how to evolve and grow the Webmaker project.  Mark has been writing about the plan for 2013.  I’m mainly thinking about ideas for tools, products, and how to do work like this at the scale of Mozilla, especially when it comes to large scale community involvement.

One of the ideas I’ve been presenting to the team is that in order for our tools to get better, and in order for a lot of people to not only want to use them, but love using them, we have to use them ourselves–we have to dogfood them.  To that end, I’ve been working on using some of our tools these past two weeks.

Today I decided to try and build something that Mark’s been discussing a lot: Popcorn-powered slideshows that tell a story in time using images.  Popcorn Maker is capable of doing something like that, but capable isn’t the same as appropriate.  Yesterday I noticed a post on reddit about the images sent with the Voyager spacecraft into deep space.

Here was a perfect opportunity to test Mark’s theory.  What if, instead of a page of static images you scroll through, I made a visual slide-show using audio and video?  I’ll skip to the end and share the final product with you, before discussing what the dogfood tasted like.

Overall, I’m pleased with how it turned out.  I think it does something more than a web album can–I get teary watching it, to be honest.  I also think it was harder than it needed to be.

In our planning meetings, I’ve said many times that our tools need to level-up.  When I say this, it seems to get some people’s backs up.  On one level I understand that–doesn’t it mean you’re attacking me if you say what I made isn’t good enough?  On the other hand, I’m so used to this way of working from my time developing in the Mozilla and Firefox communities, where we all know that our code has bugs, lacks features, needs to get better, that I’ve stopped being offended when someone offers a critique like this: “File a bug.”  It shouldn’t be personal.

So to prove that I mean no disrespect, and that I can dish it and take it, allow me to tell you how much work there is to do on Popcorn Maker, which I just spent a good part of a year building:

• There’s no way to specify that I want to cover the entire video area with an image or colour.  For my purposes, I wanted to black out the video at a certain point, and was forced to put a Text plugin, with black text, black background, and have it fill the entire video area.  Hacky.
• There’s no way to say, “Use this list of images” and have Popcorn Maker uniformly spread them over a time period.  I had to manually calculate how long to show each image in order for all 116 to fit in the time I had.  It’s finicky to do with the mouse, laborious when entering time codes manually.  I had to do both.
• When you’ve arranged a whole bunch of equal-spaced images, there’s no easy way to go back and say “Let’s show all these for 1s instead of 1.6s each.”  We sort of support multi-select, but doing it on 116 images over 4 minutes is impossible.  I’d like some way to operate on that group.  Maybe the super scrollbar can give me some better elastic-band style highlighting?
• I really wanted a way to be able to drag a plugin to a new start time, then enter a duration vs. an end-time, that is, be able to say “+1.6″ and have it figure out what that end time is.
• I wished I could have lengthened the duration of the fade in/out, as I needed a slower pace for the affect I was trying to get.
• Having to set the exact same settings each time I drop a new plugin on the timeline is annoying.  Why not use the same font, colour, sizing, placement, etc. as I did a second ago on the last one?
• When I tried to remix what I’d made, and swap in a new video, all my quotes and double-quotes had been turned into HTML escape sequences.  Annoying.

I could go on.  These are all bugs or feature requests, and I simply need to file them.  However, they are bugs and feature-gaps I’m now aware we have in our tool.  That’s a really humbling, and important realization for a developer.

“How do we know if something’s better?” is another question I’m hearing a lot.  Well, in my case, one real data-point that we could measure here is how long it takes to make something like this.  I think it took me about 3-4 hours to make this with Popcorn Maker.  That’s not terrible, and it’s not ideal.  “Can we make it easier, faster to build a web-based slide show?” is a question I’m asking right now.  I think ‘yes’.  For one thing, we should look into automatically connecting a bunch of images to a timeline, perhaps without having to actually position each one.  What if you specified that the slide show was going to occupy a certain time, and let the tool figure out which image gets shown when, and for how long?  I would have loved that here!

Dogfood doesn’t taste great.  In fact it’s easy to let that foul taste keep you from doing it at all.  Nor does it feel good when someone else tries to use your thing and finds it lacking in a whole bunch of ways.  However, if you want to get to the point where your software is fun to use, you have to admit that it has issues, make plans for improving things, and then execute on that plan.

I’m really pleased that I was able to use nothing but the web to build this.  I think there’s still a bunch of things to do to make it better, but it’s nice to not be starting at zero.  I’m looking forward to improving all our tools, and bringing more web makers into Webmaker.

December 05, 2012 09:05 PM

‘Webmaker is both a product and a community.‘ This is the conclusion that a bunch of came to last week as we were looking at goals for Webmaker 2013. We need a product that delights, gives people value and builds up demand for content that could only be made on the web. We also need a global community of people excited to teach about the open tech of the web and the creative freedoms that it offers We need to build both of these things.

Based on this discussion and many others, I’ve worked up a first cut description of what Webmaker 2013 might look like. It includes: a product and community description; audience definitions; goals and objectives; and top level metrics. Over the next few weeks, I want to discuss these more broadly and then refine them.

The idea that we want to make a product that people love is at the core of Webmaker 2013. With that goal in mind, I’m proposing a number things that change what we’ve made so far:

• The product will be positioned as a way to ‘animate the web’. Popcorn-powered videos, slideshows, etc. have caught people’s attention. We will make these our core differentiating feature.
• Thimble + Popcorn Maker + Xray Goggles will become more tightly integrated. Your Popcorn video will come wrapped in a Thimble paged editable with Goggles.
• Webmaker.org will become a) showcase for best content people make and b) jumping off point for remixing and learning.
• As part of this, we’ll make flexible gallery tools for ‘me, my friends and my themes’. The galleries themselves will be highly hackable.
• Also: we will start to look like a distributed social network, pushing your content into Tumblr, Twitter, Pinterest, Facebook and always including a remix button that pops you out to Webmaker.org so you can learn, tinker and (re)create.
• Hive + Code Party will merge into an ongoing global community of mentors with local roots. This will be core to the movement building side of Webmaker.

Assuming we go in this direction, we’ll need to again evolve the way we describe Webmaker to the world. Long time Mozilla engineering genius Johnathan Nightingale has been known to day that ‘Firefox combines both rocket (awesome browser) and payload (user choice and web standards)’. We could think of Webmaker in a similar way:

Rocket: Apps to author web pages that move: videos, slideshows, etc. that combine content and code from across the web. The tools solve a problem: they make it way easier than it is today to mix your phone, web and social media content together into a compelling, moving collage that you can share with friends. Also, the content that pops out the other end is magnetic, edgy, useful, new. It looks unlike anything people are making today because it’s made by combining real, live and, sometimes, constantly changing content from across the web. People will love this stuff. And no one else has it (yet).

Payload: Ultimately, this gets people to expect a remix button for everything. People start by making videos, slideshows, etc. that could only be made with the open technology of web. The videos, etc. pull material via URLs + APIs. They pull from your phone, your social networks, everywhere. They make it easy to see, edit and drop in code. Over time, people realize Webmaker content is remixable, view sourceable and can change as the web changes. Also, the tools and the content you make show you how the web works as you make things. There are ‘remix’ and ‘how to’ buttons on every piece of content created using Webmaker tools.

These tools can be hugely popular. I believe that quite deeply. But, like Firefox, with its millions of early adopters who installed a new browser on a friend’s computer, we also need a community and a ground game. I propose that we leverage our existing work on Hive and Summer Code Party to build a global community of mentors, teachers, techies and evangelists. It might be described this way:

A global community of makers and mentors excited to show people what you can do using the creative and technical freedoms of the web and (and open tech in general). They use open tech and a maker attitude to teach everything from art to science to citizenship. Sometimes, they use Webmaker tools. Sometimes they use Scratch. Sometimes they hack with toys and hardware from the junkyard. And everything in between. This community is built on the event and local learning network models that we’ve begun to develop with Summer Code Party and HiveNYC.

The important thing about this community is that, like Mozilla Festival, it’s not just about our tools. It’s about something bigger: using the maker spirit to teach and inspire. Mozilla has an important role to play in connecting this maker spirit back to the web and showing how you can live an online life that taps the creative and technical freedoms of the web to their fullest. This Mozilla side of making — and the Webmaker products — should both fuel and draft in the wind of the broader maker movement, just as Firefox did with open source a decade earlier.

As I say, all of this thinking — plus the detailed goals and objectives I’ve written up for Webmaker 2013 – is a draft for feedback. I’ve set up a bunch of threads on the Mozilla Webmaker mailing list to discuss different aspects of this plan. That’s the best place to go if you want to join into a discussion on these ideas. Of course, comments here on my blog are also welcome.

December 04, 2012 11:42 AM

Intel and Lenovo launched the first Android device based on the Intel Atom processor in 2012 at CES. Some of the current Android x86 devices include the Lenovo K800, the Orange San Diego, the ZTE Grand X IN, and the Motorola RAZR i.

Mozilla has been internally testing Android x86 builds of Firefox. Firefox runs really well on the Motorola RAZR i. Once we get Android x86 nightly builds stood up and running automated tests, we’ll start planning for a release.

In the meantime, you can try a developer build of Firefox for Android x86 yourself. Just like the ARM version, Firefox runs on Android 2.2 or higher. It should run fine on the Lenovo K800, which shipped with Android 2.3.7 (Gingerbread).

WARNING: This is not an officially released version of Firefox. It will only run on Intel-based (x86) Android devices.

If you don’t have an Android x86 device, you can still try the build using the Android x86 emulator. It’s the same as standard ARM emulator, but since it’s x86, it runs a lot better on your PC. Just make sure you update your Android SDK so it has the latest AVD images, and make an AVD using the x86 image. Also make sure you enable Use Host GPU as well:

Please let us know how the build works on your Android x86 devices and remember to file bugs.

November 28, 2012 04:31 PM

We started working on Mozilla Webmaker about a year ago with this thesis: inviting people to make, tinker and share things is the best way to teach the world how the web works.

Much has changed with Webmaker since. We’ve evolved Popcorn Maker into something lean and intuitive. Thimble has emerged from the Xray Goggles. And our vague ideas about community building turned into a worldwide Summer Code Party. But one thing has stayed constant: our core belief that making is the fastest (and funnest) way to learn things.

We’re not the only people who think that making and learning are deeply intertwined. Seymour Papert thought it 30 years back. Today, you see it in action at Maker Faires and in every hackerspaces around the world. You see it in the hands on side of the ‘learn to code’ movement. And, as demonstrated in NESTA’s recent report on innovation in education, you see it in discussions that link the resurgence of making, hacking and craft with new ways to teach and learn.

Reflecting on all this recently, I was reminded that the opportunity here is is much bigger than teaching the world the web. In a world where digital things make making easier and cheaper than ever, we have the chance to move the learning ball quite far in at least three areas: digital literacy; digital citizenship; and STEAM.

If you think about a Maker Faire or a hackerspace for a moment, you can see a snapshot of this opportunity. You turn one direction: you see people who have taught themselves to engineer robots, rocket ships and all kinds of fantastical contraptions. You turn another: you see a table full of kids and parents teaching each other to code with Scratch. And, all around you: you see people helping each other, tinkering, collaborating and inventing by doing. These are all things that we tend to tap into more as we become citizens of the web.

Also: appetite in making and digital creativity is growing. Mozilla recently asked UK young people aged 8 – 15 about coding, making games and making web pages: 67% said they wanted to know how to do these things yet only 3% said they already had these skills. In the same survey, UK parents ranked coding the most important thing for their children to learn after science, english and math. Also, consider that a billion people on our planet already post or curate content online: these people may not be full fledged digital makers yet, but they clearly comfortable creating and sharing online. My guess is that many of these people just need a little nudge to dive deeper into hacking, coding and making.

Of course, tapping into making as a way to make substantial progress on digital literacy, digital citizenship and STEM would require more than just MakerFaires, Popcorn and social networks. It would require a concerted effort to build products, run programs and raise public awareness in a way that gets 100s of millions of people excited about the connection between making and learning. The thing is: it feels like many of us are already making this concerted effort. What we’re not doing yet is talking to each other or telling the world how huge this opportunity is.

As I start planning for Webmaker 2013, I want to find ways to fix this: to build a bigger making and learning agenda. We already work with many people are building this agenda in their own areas: Tim O’Reilly and Dale Dougherty at Make; Tom Kenyon and the people at NESTA in the UK; Connie Yowell and the team at MacArthur. We all agree that the coming year is the right time to really push the idea of making and learning globally. As a first step, I’m working with some of these people on a simpler way to describe making + learning. Let me know if you want to be involved.

November 26, 2012 05:54 PM

It’s November, which means it’s planning season. Last year at this time we developed a plan for our first year of Mozilla Webmaker. Our overarching goal for 2012 was:

Roll the best Drumbeat software and learning resources into a cohesive webmaker offering

As outlined in slides from our October board meeting, we met this goal — and we also learned a great deal about what we mean by Webmaker and how we need to evolve it as a product.

Building on this first year, our overarching ambition for 2013 is to:

Grow Mozilla Webmaker into a product that people love.

We’re in the early stages of developing a more specific set of goals and objectives for 2013. I outlined a first cut at these on last week’s Webmaker call:

External facing:
In 2013 we want Mozilla Webmaker to be:
1. A popular way to make and remix web content
2. That levels up your web skills as you make things
3. Powered by a global community of makers and mentors

Internal facing:
In order for this vision of Webmaker to succeed we need to:
4. Create a culture of excellence (process + outcomes)
5. Tee up solid long term funding and resource base
6. Establish a regular flow of new product ideas

We’ll be honing these over coming weeks. Expect posts with reflections and updates from myself and others on the Mozilla Webmaker team. In the meantime, I’d love to get reactions on this first cut at top level goals.

November 25, 2012 03:02 PM

For a while now, I’ve been having problems with my Firefox profile. To be fair, it’s mostly because of random about:config tweaks I’ve made, but still, not being able to test the new SocialAPI stuff was pretty annoying. So I decided to try resetting my profile, to clear out all the junk, and hopefully even make it a little faster.

But, as the page I linked to just up there mentions, resetting your profile will lose your open tabs, windows and tab groups, which kinda sucks, because I have 57 open tabs, in various groups, and I really don’t want to lose them! Fortunately, I’m a programmer, so I hacked on Firefox to get it to save and restore my tabs, and now I’m a happy camper!

A couple of days later, one of my co-workers had some similar problems, and also wanted to re-set his profile to try and fix them. I hadn’t saved the results of my hacking, so I had to re-create it for him from a combination of memory and the documentation. The new code I came up with looked something like this:

var x = gBrowser.tabs
var rv = "var tabs = [\n"
for (var i = 0; i < x.length; i++) {
  rv += '  "' + x[i].linkedBrowser.contentWindow.location + '",\n';
}
rv += '];\nfor (var i = 0; i < tabs.length; i++ ) {\n'
rv += '  gBrowser.addTab(tabs[i]);\n}\n'

To run it, first open about:config, and make sure the devtools.chrome.enabled preference is set to true (double-click it if it isn’t, and it should switch automatically), then go to the Tools » Web Developer » Scratchpad menu item, which should open up a small new window with some javascript comments in it. While that window is focused, click on Environment » Browser, to make sure that you’re running the code in the browser’s chrome (instead of in the page’s content). Paste the code in, and click Execute » Display.

That should result in a bunch of code in grey surrounded by /* and */ that looks like:

var tabs = [
"http://weblog.latte.ca/blake",
"http://breakingtheegg.tumblr.com/",
];
for (var i = 0; i < tabs.length; i++ ) {
  gBrowser.addTab(tabs[i]);
}

Copy that out of the scratchpad into your favourite editor, remove the /* and */, and run the profile reset.

Once you’re done resetting your profile, you’ll need to change the devtools.chrome.enabled preference to true again, and then re-open the Scratchpad, paste the new code you saved back in to it, click on the Execute » Run menu item, and shazam! All your tabs should be back (although they won’t be in their original tab groups. If anyone needs me to figure out how to do that, just let me know, and I’ll give it a try).

November 23, 2012 06:42 PM

With the majority of the webvtt parser now written (untested, unrevewied, but written), we’ve turned our attention back to testing.  Previously the class wrote 300+ validation tests.  These were simple VTT files that allowed us to check that a parser (or validator) correctly passed or failed a particlar VTT file, each one designed to exercise a different aspect of the spec.

What I wanted to do next was convert all these files into unit tests.  We decided to try and use node-ffi so we could write the tests in JS instead of C.  This turned out to be a better idea than reality, and left us with innumerable issues wrapping our code (especially the parse callback), build issues on Windows, etc.  Undaunted, I suggested we try Python’s ctypes, hoping it would be more mature.  This was better, and we were able to make progress; but it added a level of complexity that I don’t think was helping the majority of students.  If you’re struggling with C code, and your professor hands you a half-finished Python wrapper and says “write tests for the C code,” it’s a conceptual leap that’s hard to make.  So I abandoned that, too.

I tell my students to try things and not panic when it fails, so I have to do the same.  At this stage I was looking for a solid solution vs. something hacky that might fail a third time.  In the end I followed Ted’s suggestion of using Google Test (gtest), which he’s often praised from his work on Breakpad.  The students are (mostly) familiar with JUnit from other courses, so the xUnit style of Google Test is perfect.  Furthermore, the docs are awesome, it works cross-platform (and on TravisCI), there’s plenty of examples, and lots of other projects using it, which the students can read.  Win!

I added gtest to our autotools build (this template on github is helpful, if you ever have to do the same), and now TravisCI runs all our unit tests and reports back to github for any pull requests people make.  In order to test our C code with gtest, Caitlin wrote a C++ wrapper, and Rick wrote a test fixture class, allowing our tests to become very simple.  Here’s an example:

/**
* Verifies that the parser correctly parses a "vertical" key, followed by U+003A ':',
* followed by 'rl' (indicating that the text be positioned vertically, and grows towards the left)
*
* This cue should have a Vertical orientation with direction RightToLeft
*
* From http://dev.w3.org/html5/webvtt/#webvtt-vertical-text-cue-setting (09/28/2012):
* 1. The string "vertical".
* 2. A U+003A COLON character (:).
* 3. One of the following strings: "rl", "lr".
*/
TEST_F(CueSettingVertical,RL)
{
  loadVtt( "cue-settings/vertical/rl.vtt", 1 );
  ASSERT_TRUE( getCue( 0 ).isVerticalRightToLeft() );
}

This loads uses the cue-settings/vertical/rl.vtt file, parses it, and makes sure there is 1 cue returned.  It then makes sure that the vertial-right-to-left attribute is true.  Here’s what the file looks like:

WEBVTT

00:00.000 --> 00:10.000 vertical:rl
Payload

Great, right? Even better, our first dozen tests have already found a bunch of bugs in the implementation code. It’s nice to be able to instantly show the students the benefits of writing lots of little tests. They also had to quickly learn how to manage an automated build and testing system with the desire to write and land a lot of tests. What do you do with tests that are known to case things to fail or crash?

Google Test provides a way to disable a test without removing, or commenting it out–you simply use the DISABLED_ prefix in the test’s name.  When you run your tests you get a report that there are disabled tests:

[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from CueSettingVertical
[ RUN      ] CueSettingVertical.RL
[       OK ] CueSettingVertical.RL (11 ms)
[ RUN      ] CueSettingVertical.LR
[       OK ] CueSettingVertical.LR (0 ms)
[----------] 2 tests from CueSettingVertical (11 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test case ran. (11 ms total)
[  PASSED  ] 2 tests.

  YOU HAVE 5 DISABLED TESTS

Running these disabled tests is possible with a command-line flag: --gtest_also_run_disabled_tests. Debugging a failing disabled tests requires some more work. Google Test is pretty good at not blowing-up when a test throws or segfaults. To catch such errors in the debugger, you also need to pass the --gtest_break_on_failure flag to your unit test executable. This let’s gdb deal with the crash. For example:

$ cd webvtt
$ ./configure --enable-debug
$ make && make check
$ cd test/unit
$ gdb payloadnode_unittest
GNU gdb 6.3.50-20050815 (Apple version gdb-1515) (Sat Jan 15 08:33:48 UTC 2011)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin".
(gdb) run --gtest_also_run_disabled_tests --gtest_break_on_failure
Starting program: /Users/dave/Sites/repos/webvtt/test/unit/payloadnode_unittest --gtest_also_run_disabled_tests --gtest_break_on_failure
Reading symbols for shared libraries ++. done
Running main() from gtest_main.cc
[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from PayloadNodeTest
[ RUN      ] PayloadNodeTest.DISABLED_NodeCount

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000008
0x000000010001247d in WebVTT::NodeFactory::createNode (otherNode=0x0) at nodefactory.cpp:8
8		if ( WEBVTT_IS_VALID_INTERNAL_NODE( otherNode->kind ) )
(gdb) bt
#0  0x000000010001247d in WebVTT::NodeFactory::createNode (otherNode=0x0) at nodefactory.cpp:8
#1  0x000000010000b2d6 in WebVTT::Cue::nodeHead (this=0x100200ac0) at cue:99
#2  0x000000010000b2fa in PayloadNodeTest::getHead (this=0x100200ba0) at payloadnode_testfixture:11
#3  0x00000001000088d5 in PayloadNodeTest_DISABLED_NodeCount_Test::TestBody (this=0x100200ba0) at payloadnode_unittest.cpp:13
#4  0x0000000100026433 in testing::internal::HandleSehExceptionsInMethodIfSupported (object=0x100200ba0, method={__pfn = 0x21, __delta = 0}, location=0x100035fe3 "the test body") at gtest-all.cc:3413
#5  0x000000010003155f in testing::internal::HandleExceptionsInMethodIfSupported (object=0x100200ba0, method={__pfn = 0x21, __delta = 0}, location=0x100035fe3 "the test body") at gtest-all.cc:3449
#6  0x000000010001d9b1 in testing::Test::Run (this=0x100200ba0) at gtest-all.cc:3485
#7  0x0000000100024748 in testing::TestInfo::Run (this=0x100200090) at gtest-all.cc:3661
#8  0x000000010002489b in testing::TestCase::Run (this=0x1002004f0) at gtest-all.cc:3768
#9  0x0000000100024b9d in testing::internal::UnitTestImpl::RunAllTests (this=0x1002001c0) at gtest-all.cc:5591
#10 0x00000001000268e5 in testing::internal::HandleSehExceptionsInMethodIfSupported (object=0x1002001c0, method={__pfn = 0x100024930 , __delta = 0}, location=0x100035cc8 "auxiliary test code (environments or event listeners)") at gtest-all.cc:3413
#11 0x0000000100030fe8 in testing::internal::HandleExceptionsInMethodIfSupported (object=0x1002001c0, method={__pfn = 0x100024930 , __delta = 0}, location=0x100035cc8 "auxiliary test code (environments or event listeners)") at gtest-all.cc:3449
#12 0x000000010001d161 in testing::UnitTest::Run (this=0x10005d280) at gtest-all.cc:5226
#13 0x00000001000128f6 in main (argc=1, argv=0x7fff5fbff790) at gtest_main.cc:37
(gdb)

Here I’ve compiled the code and tests in debug mode, which will produce an executable gtest program I can run for my unittest named payloadnode_unittest. From within the test/unit directory, I run the program under gdb. I tell gdb to run the program with the appropriate flags for testing. It happily runs the tests, including my DISABLED_ tests, and promptly crashes, dropping me back into the gdb prompt. Here I can ask for a backtrace using the bt command, and a quick look at the stack shows me that we’re calling WebVTT::NodeFactory::createNode with a null node from WebVTT::Cue::nodeHead. Now I know where to start digging in order to fix this.

As we wind down the semester, we’ll churn through all these tests and fix the bugs they find. I’m glad to finally have a working testing infrastructure we can all rely on.  Once we have the tests in shape, and bugs shaken out, we can move onto integration in Firefox and the track element.

November 23, 2012 05:25 PM

A key part of Mozilla and the Firefox product is that it was built by a huge team of volunteers. As Mozilla has hired more full-time engineers, we haven’t necessarily grown our volunteer hacker community to match. In order to build our volunteer community even more, I am now hiring a full-time community manager for Firefox engineering.

I am very excited about the opportunity to have a person whose sole job is dedicated to understanding, growing, and nurturing the Firefox developer community. For a while we’ve had a great “Coding Contribute” team who have been creating dashboards and tools in their spare time, and individually welcoming new contributors. This position will expand on that team and make it more effective. There are many things we don’t understand about our volunteer community, and it will be awesome to have somebody who can spend their time understanding volunteer motivations, skills, limitations, joys, and frustrations. The community manager will be an essential resource for module owners, managers, and other team members to understand how to bring in and guide volunteers and how volunteers can help meet Mozilla’s goals.

Anyone who is interested can view the job description and start the application process here. I would like to especially encourage current members of the volunteer community to apply, if this is something that you are passionate about and are interested in a full-time job with Mozilla. If you aren’t sure and you’d like to talk to me about it before applying, please feel free to contact me.

November 19, 2012 06:45 PM

Last night as a few of us were working on the webvtt code, Dale popped into channel to ask if others were having trouble building.  After a few minutes we concluded that the tree was indeed broken, and that the last commit had failed to add some header files, update Makefile.am, etc.  This kind of thing happens all the time, and it’s the reason that smart developers have learned to never trust themselves, and instead rely on continuous integration.

Some months back I tasked Michael with getting our project building in TravisCI.  If you’ve never used it before, TravisCI is not unlike Mozilla’s Try-Server.  By adding a git hook to your project, a config file to the root of your source tree, and wiring Travis and Github together, you get per-commit, clean-room builds of your code.  Better still, you get this integrated into pull requests, such that anyone wanting to merge is going to see (and show others) that the build is working or failing.

I spent a few minutes integrating what Michael had done into my repo last night so we wouldn’t burn the tree down again.  I’ve added a build status icon to our README.md, which points at our Travis build page.  Anyone doing pull requests from now on will get live feedback from Travis/Github about the state of the build in gcc and clang on Linux.  Once we get the test harness integrated we’ll add tests to this too.

I’ve set it up to notify us on IRC, but there is a bug in Travis which seems to affect irc.mozilla.org.  Hopefully that will get fixed soon so we can see build status there.

I continue to be impressed with TravisCI, and would highly recommend it for your project.

November 17, 2012 04:40 PM

I was working on a project today and was seeing weird sort behavior and couldn’t figure out what was going on for a while. I finally discovered that JavaScript Array.map wasn’t returning the values I expected. In particular:

> "11.4.402.265".split('.').map(parseInt)
[11, NaN, NaN, 2]

I was a little worried that Mozilla was broken, so I asked the JS experts in IRC and got this interesting diagnosis from Jeff Walden (Waldo):

parseInt takes a second parameter that’s a radix; map calls its function with (value, index, thisArray)

This magic behavior of Array.map can be useful if you want to do even-odd behavior or other tasks, but it happened to be my personal footgun today. Be careful! My code now interposes its own function:

a.split('.').map(function(i) { return parseInt(i); });

I’m not sure I really want to know why parseInt(11, 0) returns 11.

November 15, 2012 07:47 PM

We just launched Popcorn Maker 1.0 this weekend at Mozilla Festival 2012, and while the focus of the app is obviously web video remixing, one of our favourite features as a dev team was the JavaScript Crash Reporter.  Bobby and I wrote a little bit about it in our Mozilla Hacks post, but I wanted to say some more about the potential for this technique.

Working on Firefox code, I’ve long been a fan of the Mozilla Crash Reporter, which uses Google’s crash reporting system, Breakpad, and the Soccoro server.  Typing about:crashes in Firefox’s address bar will show you any crashes you’ve logged, which are links to crash-stats.mozilla.com.  Here’s an example.  When you’re lucky you get a stack showing where things went wrong, info about bugs related to this crash, aggregated data for other users’ crashes in the same spot, etc.

For developers it’s an incredible source of information, since often these crashes occur out in the wild, and under circumstances the developers can’t reproduce.  Crash reporting in native apps isn’t new, of course.  OS and application vendors routinely use this data-driven debugging method to gather useful data about edge cases in their code, and to make corrections.

The technique isn’t used as often in JavaScript applications, though.  More often than not, a user’s contribution to the data of a web app is about marketing or tracking vs. improving development and stability.  Another reason is that many pieces of JavaScript are libraries vs. apps, and get used on more than one installation, making it undesirable that code phone-home to report an issue.

With Popcorn Maker we wanted to make it easy for our users to give feedback and report issues.  Because the project isn’t for developers, asking that they find and file bugs in our issue tracker wasn’t realistic.  So we made a Feedback button and form, and also an automated Crash Reporter to deal with top-level exceptions (i.e., those that can be caught using window.onerror).

We’ve been using these techniques now for about about a month, since the beta launch for Ryan’s TED talk on Popcorn, and through the 1.0 launch, and I can now talk about what it’s like using them.  First, and perhaps not surprisingly, people don’t tend to leave much feedback.  It takes time to write free-form comments in a form, and most people don’t want to bother.  So far we’ve gotten between 3 and 6 a day.  The quality of what people say is usually quite high, though.  When they have bothered to do the work of filling out the form, they’ve taken care to do it well, and usually give us ideas for new features.

The crash reports are another story.  This morning when I sat down to check my email, I had over 50 waiting for me.  I was able to file 8 new bugs (many of them were the same issue hit by different people).  I was also able to see that we’ve regressed one bug we thought we’d killed completely, that we’re having issues in the Sogou and Zune/Media Centre browsers, that we have a timezone related bug in some of our date parsing, and that I finally have better data about a crash we first saw last week (i.e., different browsers report the error differently, giving us a more complete picture of what is actually failing). That’s data from just a single day!

I can’t emphasize how drastically this technique has changed our development practices.  In the period between 0.9 and 1.0 we found and fixed 12 bugs that our team couldn’t reproduce (there are another 12 on file that we’re still trying to fix).  We put a lot of effort into automated and manual testing, and work hard on cross-browser testing, but there is no way for us to hit all these issues.  Furthermore, there’s no way people would file all these issues.  What would they say?  “It didn’t work” isn’t helpful for them or us.  We need to know what failed so it doesn’t happen again.  Rather than having a web app that just silently fails, we give our users the chance to become part of the development process by submitting the crash data, and a lot of them do:

You can try simulating a crash by going to http://popcorn-dev.rekambew.org/templates/basic/ and doing `Butter.app.simulateError()` in your console.  If the user wants to add extra info, they can (most don’t), and then if they click ‘Yes’ we send anonymous data about the error.  For example:

• Which version of our code they were using
• Which media URL(s) they tried
• Data from the onerror handler–message, filename, line number
• States the app had hit before the crash (i.e., which events we’d processed internally, like mediaready)
• Any DOM nodes that we tried to get and were null

We take that data and store a JSON report on Amazon S3.  Here is an example:

{
    "date": "Tue Nov 13 2012",
    "message": "TypeError: 'undefined' is not a function (evaluating 'tracksContainer.container.getBoundingClientRect()')",
    "appUrl": "https://popcorn.webmaker.org/templates/basic/?savedDataUrl=ted.json",
    "scriptUrl": "https://popcorn.webmaker.org/src/butter.js",
    "lineno": 9805,
    "mediaUrl": [
        "https://www.youtube.com/watch?v=0g2WE1qXiKM&butteruid=1352812826605"
    ],
    "stateList": [
        "trackeventadded",
        "trackeventadded",
        "trackeventadded",
        "trackadded",
        "trackeventadded",
        "trackadded",
        "trackeventadded",
        "trackeventadded",
        "ready",
        "mediaready"
    ],
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/536.26.17 (KHTML, like Gecko) Version/6.0.2 Safari/536.26.17",
    "popcornVersion": "v1.3-44-g05ac7ef",
    "butterVersion": "v1.0.4",
    "nullDomNodes": ".media-status-container, .youtube-subscribe-embed, [data-tooltip]",
    "comments": ""
}

There are a few gotchas to be aware of:

• You need to filter your errors by filename so you don’t crash when trivial things happen in scripts you include.  In our case, we use a lot of third-party APIs which can spam the console with errors that aren’t actually a problem–the YouTube iframe tries to read values in the parent document and fails with cross-origin errors.
• Firefox has an annoying bug where errors in simulated events get eaten, and don’t make it up to the top-level, see https://bugzilla.mozilla.org/show_bug.cgi?id=503244

There are also other things I’d love to figure out how to do better:

• How should we deal with minified code?  For now we’ve left our code unminified in order to get better data about what the failure was (e..g, function/var names vs munged names), and real line numbers.  It would be great to do something with source maps or the like
• How to get a stack?  By the time onerror happens, the stack is unwound, and we don’t have good data about the full issue.  Sometimes the top-level error message is useful, sometimes it isn’t.  Knowing what happened before that would be great.  I’ve thought about writing wrappers around our major module blocks to catch exceptions, cache the error (and stack if there), then rethrow.  The crash reporter could look up the last error(s) in the cache and add that data.
• It would be great to add some kind of assert API to the reporter.  Imagine if you could assert not to the console, but to the reporter, and if there’s a crash, send along failed assertion data.

I’d also love to not have to build a full UI for working with the reports   I think it would be great if web app developers could use Mozilla’s Socorro infrastructure to gather and analyze crashes.  Imagine if part of doing a Mozilla Market Place app was getting access to that infrastructure?  Chris Lonnen and I discussed this idea at the Festival.

At some point I’d like to rip the crash reporter out of Popcorn Maker and turn it into a stand-alone repo others can use.  The code is here, if you’re interested in seeing how we do it.  I’d highly encourage other web app developers to go this route.  It’s been a game changer for our team and app.

November 13, 2012 05:38 PM

Today, Mozilla Research publicly announced the Shumway project. Shumway is JS/HTML library which displays SWF (Flash) content entirely using open web technologies.

Live demo below: Click the car and then use the arrow keys to drive.

View a live demo of Shumway in action!

I am very excited about this project. There is a lot of Flash content on the web, and Flash is not available for many mobile users, including most new users of Firefox for Android and Firefox OS. Many popular sites are already converting to HTML content, but there is always going to be a long tail of Flash content which is not actively maintained. Although it’s still a research project, my hope is that Shumway will some day be able to play enough Flash content that we could include it as part of Firefox on mobile platforms.

Another possibility of the Shumway technology is that website authors could use Shumway as a JavaScript library to display their legacy Flash content in any modern browser. For example the Shumway team already has an online example of SWF content running in the browser without any browser support at all. This example works in Chrome, and we can probably make it even easier to automatically convert <embed> nodes using shumway with a little bit of scripting. (The example doesn’t work in IE 9 because IE 9 doesn’t support JavaScript typed arrays or “const”. Maybe future versions of IE will join the modern era.)

Mozilla is actively looking for volunteers who are interested in helping build out Shumway to its full potential. We need help implementing important builtin objects, testing and tuning performance, and building out a test suite. For more information, clone the github project, check out the wiki, join the mailing list, and join us on irc.mozilla.org #shumway.

November 12, 2012 09:37 PM

I complained some time ago about autoconf 2.13 failing because it assumes implicit int. People rightly pointed out that newer versions of autoconf don't assume that anymore. But new versions still come with their own cornucopias of pain. Libtool, for example, believes that the best route to linking a program is to delete every compiler flag from the command line except those it knows about. Even if you explicitly specify them in LDFLAGS. Then there's this conftest program that I found while compiling gawk:

/* Define memcpy to an innocuous variant, in case <limits.h> declares memcpy.
   For example, HP-UX 11i <limits.h> declares gettimeofday.  */
#define memcpy innocuous_memcpy

/* System header to define __stub macros and hopefully few prototypes,
    which can conflict with char memcpy (); below.
    Prefer <limits.h> to <assert.h> if __STDC__ is defined, since
    <limits.h> exists even on freestanding compilers.  */

#ifdef __STDC__
# include <limits.h>
#else
# include <assert.h>
#endif
#undef memcpy

/* Override any GCC internal prototype to avoid an error.
   Use char because int might match the return type of a GCC
   builtin and then its argument prototype would still apply.  */
#ifdef __cplusplus
extern "C"
#endif
char memcpy ();
/* The GNU C library defines this for functions which it implements
    to always fail with ENOSYS.  Some functions are actually named
    something starting with __ and the normal name is an alias.  */
#if defined __stub_memcpy || defined __stub___memcpy
choke me
#endif

int
main ()
{
return memcpy ();
  ;
  return 0;
}

…I think this code speaks for itself in how broken it is as a test. One of the parts of the compiler pass involved asserted due to memcpy not being used in the right way, crashing the compiler. Naturally, this being autoconf, it proceeded to assume that I didn't have a memcpy and thus decided to provide me one, which causes later code to break in spectacular bad function when you realize that memcpy is effectively a #define in modern glibc. And heaven forbid if I should try to compile with -Werror in configure scripts (nearly every test program causes a compiler warning along the lines of "builtin function is horribly misused").

The saddest part of all is that, as bad as autoconf is, it appears to be the least broken configuration system out there…

November 08, 2012 01:04 AM

I already blogged about support for remote debugging in Firefox for Android. Now we have support for remote Web Console too!

The remote console, like remote debugging, requires using Desktop Firefox to connect to Mobile Firefox over USB using ADB. Remote console is supported in Firefox 19 and newer releases. Here’s a quick guide to getting started:

1. USB Connection: Connect your Android device running Firefox to a host machine running desktop Firefox using USB. In a terminal, forward the TCP connection using: adb forward tcp:6000 tcp:6000
2. Firefox Desktop: Use about:config to enable remote debugging. Set the “devtools.debugger.remote-enabled” pref to true. Restart. You should see a “Tools > Web Developer > Remote Web Console” menu.
3. Firefox for Android: Use about:config to enable remote debugging. Set the “devtools.debugger.remote-enabled” pref to true. Restart.
4. Firefox Desktop: Choose “Tools > Web Developer > Remote Web Console” and a Remote Connection dialog should open. If you changed none of the other remote debugging preferences, just use the defaults. Press OK.
   
5. Firefox for Android: You should see a remote connection prompt appear. Press OK to initialize the connection. You have 20 seconds, by default, to accept the connection. You can retry 3 times.
   
6. Firefox Desktop: Once the connection is accepted, you should see a scope prompt. The prompt is used to pick the tab, or global, scope you want to attach to the web console.
   
7. Firefox Desktop: After picking a scope, you should see a Web Console window appear.
   

Once you have the Web Console up and running, you can do pretty much everything the desktop version can do. For example, you can load a web page and monitor the network activity, including viewing the network request details:

You can also use the JS console to execute JS code in the scope of the web page running on the Android device! Pretty cool!

A big thanks goes out to Mihai Sucan, who did a mountain of work to get the Web Console remoted. Mihai also blogged about the remote Web Console and you can find some of the underlying details on the remoting API on MDN.

Happy (remote) Debugging!

October 31, 2012 05:14 AM

We’re about 3 weeks away from the Popcorn Maker 1.0 launch at this year’s Mozilla Festival in London.  Just to raise my blood pressure a bit, we’ve shipped a beta today.  It coincides with the release of an amazing TED talk by Mozilla’s Ryan Merkley about Popcorn.js and Popcorn Maker.  “It would be cool if we could work with this video using the tool he’s discussing, is that possible?”  It is

I’m really proud of the work we’ve done here, not least because so many of my former and current Seneca students have worked on it as part of class projects and NSERC funded research at Seneca’s Centre for Development of Open Technology.

A huge thanks and congratulations to:

• David Seifried
• Chris De Cairos
• Matthew Schranz
• Jon Buckley
• Scott Downe
• Robert Stanica
• Mohammed Buttu
• Daniel Hodgin

I’ll blog with more info about Popcorn Maker when we ship 1.0.

Well done team, well done.

October 19, 2012 06:59 PM

We’ve been honing our description of Webmaker recently. Partly, this is so we can explain Webmaker to the world. But it’s mostly aimed to clarifying what we’re building and who we’re building it for as we move into the next phase of development.

At a recent meeting in Toronto, Erin Knight led a set of discussions on this topic. I came out of these discussions with four big takeaways:

1. Webmaker is a peer to Firefox and FirefoxOS.

Mozilla has big priorities right now: the web on the desktop; the web in the mobile environment; and web literacy. We need to start positioning Webmaker in this context, showing how Mozilla’s three big bets / priorities all tie back the same mission.

Also, we need to make the link between the value of a phone you can re-program because it’s made from the web (FirefoxOS) and the value of knowing how the web works (Webmaker). Getting web phones into the hands of millions of skilled and creative people is the key to a next wave of innovation on the web.

2. We should describe Webmaker by simply explaining what you can make.

We need to describe Webmaker more simply and concretely. We’ve been able to say ‘Mozilla wants to create a generation of people who know how the web works and can reprogram it.’ But describing what we’re building to make this happen has been difficult. We took a shot at fixing this in Toronto:

Mozilla Webmaker: a quick way to make, remix or tweak a webpage or 
video while learning how the web works.

While this isn’t quite right yet, it opens up an important new direction: we should be explaining what you can make with Mozilla Webmaker. This creates a more tangible picture in people’s minds and helps them understand how they can engage. I’m hoping others can come up with better wording than what we have above, but based on the general approach of saying what you can make.

3. Our audience is people with something to share.

Up to now, we’ve been a bit fuzzy about who we’re targeting with Mozilla Webmaker. In Toronto, we narrowed in on ‘people how have a maker attitude and something to share’ as a core audience.

There are two pieces to this. The first is is about an approach to life: one that involves tinkering, remixing and iteration. The second is about having made something that you are proud and excited about, something that you want to share or show to other people: a picture you took; a video you made; a game you’ve modified; a big idea you’ve dreamed up. We build the needs and desires of this audience into our design process as we work on the next phase of Mozilla Webmaker.

 4. Educators are also a key audience.

During the last thee months, almost 700 people organized Mozilla Webmaker Summer Code Party events. Whether they gathered 100 people or simply brought a few friends around a kitchen table, these people have played a critical role in getting Mozilla Webmaker off the ground. And they have done so because they care about inspiring and educating others about the creative potential of the web.

Personally, I hadn’t really thought about this group as one of our key audiences before. But clearly they are. These are the first people to ‘get’ what we’re trying to do with Webmaker and to feed back in to help improve it. Like the early adopters who first installed Firefox on other people’s computers, these grassroots educators and evangelists could be the core of our global community. Over the next couple of months, we need to figure out ways to more actively help them and bring them into what we’re building.

–

These four insights aren’t particularly radical. They fit with where we’ve been going with Mozilla Webmaker for the past year. However, I do think they make it easier to explain what we’re doing. They also offer increased clarity on what we need to be building and who we need to be building it for over the next six months. Erin is going to do her own post on this aspect of the Toronto discussions, looking at how we practically pull all the pieces of Webmaker into a more cohesive offering.

September 25, 2012 03:04 PM

I absolutely love this TED talk by Julian Treasure about how sound affects people. I was especially struck by his discussion of how design/architecture can greatly affect academic success by changing the soundscape in a classroom.

Personally I am very sensitive to noise. It’s exhausting to spend time in places which have even low levels of background noise, and loud events such as amplified concerts or sporting events are often physically overwhelming. Even the Mozilla offices, with their open floor plan, are frequently distracting and make it hard to do concentrated work.

September 18, 2012 07:19 PM

Fall Teaching

It’s the start of a new semester, and once again I’m teaching the Open Source and Mozilla Development course at Seneca’s Centre for Development of Open Technology.  I love teaching this course for many reasons: it’s different every time I do it and never boring; the students and I get to work on real code together; a bunch of students get to have their first experience of what it’s like to help push the web forward (hint: it’s awesome); we get to work with some amazing people in the Mozilla community.

Evolving Thinking About Student Projects

The way I run the course has evolved over the years, and this year I’m going to try something new and a bit risky.  When I taught it for the first time, I can remember sitting in a room with Shaver at Seneca and coming up with about 15 or 20 projects for things Mozilla needed.  It was literally back-of-the-napkin planning, and it was equal parts amazing/useful projects and probably-not-worth-doing.  Mike and I had no idea what students were capable of, how big to make the projects, how much complexity, what background knowledge they could acquire in time.  As a result, we got some of it right and some wrong.  And we learned for next time.

Since then I’ve tried different approaches to the course project.  Sometimes we’ve sought out “wouldn’t it be nice if we had…” type work, other times we’ve chosen to use existing bugs; sometimes we’ve done individual projects, other times had groups of 2 or 3.  There’s no one right way to do it.

The Pointer Lock Experience

Last fall I assigned students individual bugs, and then spent the second half of the course implementing the Pointer Lock API instead of doing formal lectures.  Each class we’d go a bit further and discuss how things worked, what to try next, comb through the spec, etc.  It was a lot of fun, and a nice byproduct of the effort was that Firefox shipped Pointer Lock.

One thing I didn’t like is that not all of the students got engaged with the code as much as they could have.  About half were eager to do it, and couldn’t be stopped.  The rest saw a bunch of people already working, and decided to sit on the side lines.  I don’t blame them: the code was complicated and we went fast in order to get it done.  Also, the tasks weren’t highly parallelizable, since the spec was fairly constrained and the API small.  Another problem was that I still had each student working on other individual bugs, which meant they had to make a choice between getting their graded work done and getting involved in the class project.

This time I’ve decided to go all in on the larger class project approach.  What I liked about Pointer Lock was that it was real, mattered to the web and Mozilla, and was a strange mix of cutting-edge and non-critical tech at the same time.  I also liked that it allowed the students to form a community of knowledge together around a single spec, API, etc–when students can help one another, my experience is that they do.

When I first agreed to fix the Pointer Lock bug with my class, it wasn’t really on that many peoples’ radar.  By the end of the term I was getting regular emails from people wanting to know when it would land, and reading media reports about how Firefox was shipping it.  There’s a sweet spot in there where students feel safe to learn as they go without too many critical eyes on them, but then also have enough eyes to help them find issues and solutions.  Getting the balance right is hard, in my experience.

Toward WebVTT

Earlier this summer I went searching for another “Pointer Lock”–I wanted a new spec I could implement with my class, one which mattered, but not so much we’d be blocking releases if we took a meandering path in order to get it done.  I emailed a bunch of Mozilla folks looking for ideas, and got some great suggestions.  A few of them were too complicated, too “you’d have to be deep in Mozilla lore to even understand why this matters”, too big for the time we had, or too critical for upcoming releases.  But there was one that sounded possible, “We need someone to implement WebVTT” suggested Chris Pearce.

I’ve spent the past three years working on various projects connected to HTML5 audio and video, and I love the idea of staying in that domain for yet another project.  HTML5 audio and video continue to get more and more exciting, and I’m happy to do more to help it evolve.  Funnily enough I had another student write WebVTT support into Popcorn.js a few terms ago, so it’s nice to circle back and do the native implementation now.  Also, as a member institution of the Inclusive Design Institute, I’m attracted to WebVTT’s potential for improving accessibility for web media.

Working on WebVTT in a Mozilla Context

WebVTT (Web Video Text Track) is a file format that, when paired with the HTML5 track element, allows one to pair subtitle, caption, or other metadata about a media resource with an audio or video element.  Silvia Pfeiffer gave an excellent talk about it at Google in 2011.  Our plan is to start by writing a stand-alone implementation of the WebVTT parser and then to hook that into the track element in Firefox.  Mozilla media hacker Ralph Giles has already started work on both (parser is here, track element work is here).  There’s a long road to finishing WebVTT and track in Firefox, and it’s unclear to any of us how far we’ll get.  Part of taking on a project like this is not allowing yourself to get overwhelmed by the end goal, and instead making small steps that add up over time.

My plan is to work with the students to plan and execute this project.  Our first deliverable, due next week (get busy guys!), is a set of conformance tests for the WebVTT file syntax and parser.  Once that’s done I’ll have people split out on different paths, some people figuring out UTF-8, getting the parser passing all the tests, dealing with edge cases, etc, and others working on DOM bits in Firefox, hooking the parser into the media code, etc.

Because this is real, and because this is open source, you’re welcome to join us.  People often tell me they’d like to come and learn Mozilla at Seneca; well, you don’t have to be physically present to do it.  We’ll be working on the parser in github using Ralph’s repo, and Bugzilla for the DOM/media work (I’ll post bugs as they get filed).  The students will be in #seneca, #introduction, and #media mostly, so if you want to talk to them on irc, those are the places to do it.  Feel free to help us write code, test things, give advice, encouragement, or just say hello.  Working on something this complicated is daunting for a student, and part of the reason I know it will work is because of our community and the support available to them.

As usual, I’m somewhat terrified of the project, unsure if it’s going to work, trying to figure out how to teach all the things the students need to know in order succeed.  At the same time, after living through it in a bunch of other projects with students, I’ve come to recognize that this terror is not really a bad thing: it motivates; it excites;  it humbles.  I want to give the students a realistic experience of being out in the wild, and I can’t do that by playing it safe myself.  So here goes nothing…here comes WebVTT, implemented by a bunch of college students!

September 18, 2012 04:51 PM

People often ask: Where did Mozilla Webmaker come from? And, how does it fit into the big picture of Mozilla’s mission? There are loads of materials online that answer these questions. I figured I should create a ‘reading’ list for would be web makers that pulls together some of the main threads. Here it is.

1. The Story of Mozilla

This video is a great place to start. In 3 minutes, you get primer on how Mozilla used Firefox to keep the web alive and on where we are headed next with mobile and web literacy. The main take away: Mozilla is a global community of people creates compelling products and experiences that build openness into the internet. This is an important foundational idea to get. It’s how Mozilla thinks about itself.

2. The Mozilla Manifesto

While it could use an update, the Manifesto is still a solid foundation answering the question: what should Mozilla be working on right now? The Webmaker initiative has it’s roots in the principle that “… individuals must have the ability to shape their own experiences on the Internet.” This almost is impossible unless individuals have some basic literacy in how the web works and how to program it. That’s why Mozilla is making such a big bet on web literacy. The Manifesto also says that Mozilla should “build and deliver great consumer products that support [these] principles.” This is why we’re working so hard on Thimble, Popcorn and high quality remixable content: we believe compelling online creativity apps are powerful way to promote web literacy.

3. Creating a web literate planet

In September 2011, I started a series of blog postings outlining the basic web maker concept and asking for feedback. These posts argued that “… Mozilla has an opportunity to build the next generation of web makers.” They also set out the basic idea that we need both teach and build tools that encourage creativity, tinkering and invention on the web. These posts formed the touchstone for what we’re actually building in 2012. You can read them in order as a set or go back to ‘#nextbeat’ tag in my blog. While the #nextbeat version is confusing (you have to read from the bottom up as they are in reverse order), it’s also more interesting as it includes all the original comments and public discussion about the webmaker concept.

4. Mozilla Webmaker 2012 plans

By the end of 2012, we’d agreed to put significant resources toward what is now called Mozilla Webmaker. Our top level goal was to “… roll Mozilla’s best software and learning resources into a simple ‘kit’ for web makers.” Practically, this meant building Hackasaurus, Popcorn, Hive, Open News and many other efforts we’d started under the Mozilla Drumbeat banner into a cohesive offering and brand. I posted an overview of our plans to do this in February 2012, including links to team-specific plans. There is also a wiki page with the top level Webmaker goals and objectives approved by the Mozilla Foundation board in December 2011. As I thought experiment, I recently did a July MoFo Goals Review. While we’ve still go work to do, we’re tracking well.

5. Campus Party talk on the web and creativity

Now that we’re almost a year into the Webmaker conversation, I’ve been turning my mind back to the question: what’s the big picture strategy we need to keep the web open and vibrant? At the broadest level, I think the answer is a mix of products, literacy and public policy that bake the values of the Mozilla Manifesto into the web and into our expectations of how the web works. Mozilla’s three big projects right now — Firefox, Firefox OS and Webmaker — cover the product and literacy bases. We also need to find a way to shape policy, at least in cases where it threatens the web. I did a talk recently at Campus Party Europe that looks these things through a big picture lens. It’s rough and a bit long, but this talk is worth watching if you want to situate Mozilla Webmaker within the context of keeping the web open for the very long haul.

–

Of course, these five posts are just a primer. There are thousands of posts and reflections that people have written about the webmaker concept. And there is even more out there on the broader topic of web literacy. I’d be interested in hearing what other reading you find helpful or inspirational on this topic. If something comes to mind, please post a link as a comment below.

September 06, 2012 11:55 AM

[crossposting from mozilla.dev.platform; please continue the discussion there. Google Groups link.]

As of yesterday, Pymake has been enabled on all Firefox Win32 builders. This includes mozilla-central, mozilla-inbound, try, and all project branches (except elm). Clobber build times (this includes Try builds!) should go down by 45 minutes to an hour, resulting in faster turnaround for developers and lower load on the infrastructure.

Pymake has now become the official way to build Firefox 18 and above on Windows. If you’re on Windows and you haven't moved to Pymake yet, you should: it's really simple. I recommend setting up an alias to the in-tree Pymake in your MSYS profile.

GNU Make will still work for now, but we won't have tier 1 continuous integration for it (at least for Firefox — Thunderbird's still on GNU Make, but I’m hoping to switch those builds to Pymake soon). Win64 builds are also on GNU Make right now, but those aren't tier 1.

A few notes:

• If your project branch is seeing build failures on Windows:

  1. make sure you're tracking an up-to-date mozilla-central: the last couple of fixes landed earlier this week and more fixes will be coming down the line
  2. try setting a clobber for Win32 builds on your branch.
  3. ask in #developers or #pymake -- perhaps someone might be able to help you out
  4. if all else fails, file a bug with releng similar to this one.

• While try builds are much faster now, we also lose the ability to build Firefox 15-17 on them. I spent a bit of time investigating workarounds but didn't get anywhere, unfortunately. Update 3/9: Simply add this patch to your queue before pushing to try.

• This also means that you need to pushing an up-to-date mozilla-central to try, otherwise you will see failures on Windows.

• One out of every 20 builds or so is failing with a strange error in dom/bindings/test -- this seems to be corruption caused by a race condition that isn't being properly handled somewhere but honestly shouldn't be happening in the first place. If you see an error in dom/bindings similar to the one in this log, please retrigger that build. The error's being tracked in this bug.

I'd like to say thanks to everyone who helped push it over the finish line: ted, khuey, coop, gps, glandium, catlee, bsmedberg, bhearsum, and anyone else if I missed them (sorry!).

September 01, 2012 02:30 PM

My wife teaches high school physics. She loves putting quotes from famous physicists, mathematicians, and inventors on the wall of her classroom. After the death of Neil Armstrong, there was a lot of media coverage and I discovered a great quote from a speech that he made to the National Press Club in 2000:

“I am, and ever will be, a white-socks, pocket-protector, nerdy engineer, born under the second law of thermodynamics, steeped in steam tables, in love with free-body diagrams, transformed by Laplace and propelled by compressible flow.”

Because I thought my wife would really like this, I decided to make a poster of this quote on top of a picture of Mr. Armstrong walking on the moon. I created the design on zazzle.com and purchased it. Today I got the following response from Zazzle:

ORDER CANCELLATION NOTICE – Please modify your design(s) and re-order.

Hello Benjamin Smedberg,

Thank you for your recent order: XXXX

Unfortunately, we are unable to process your order due to a conflict with one or more of our acceptable content guidelines.

As a result, the following item(s) cannot be produced:

Title: Apollo 11 Neil Armstrong.jpg, I am, and ever wi…
Product Link: XXXX
Result: Not Approved
Content Notes:

• Design contains text or image that is in violation of an individual’s rights of celebrity/publicity. If you are interested in purchasing Official Licensed Merchandise from Zazzle please visit: www.zazzle.com/brands
• Design contains an image or text that may be subject to copyright. If you are interested in purchasing Official Licensed Merchandise from Zazzle please visit: www.zazzle.com/brands
• Your design contains an image or text that may be trademarked. This may be due to the actual design of the product, description or search tags that are associated to your product. Please feel free to submit a new design to our Marketplace from original elements

I suppose that the speech itself could be subject to copyright, but given the size of the excerpt and the occasion I don’t see how it is really relevant. The photo, being a product of NASA, is not subject to copyright. And I don’t see how right of celebrity or trademark could possibly apply to this at all.

Come on Zazzle, stop being stupid.

August 28, 2012 01:25 PM

The Mozilla Webmaker idea has come a long way. This time last year, it was just that: an idea. The Mozilla Drumbeat team met late last July to discuss where to go next. While on vacation, I wrote up notes from that meeting and began a series of blog posts about what became Mozilla Webmaker.

During this year’s vacation, I took a look back at last year’s notes. Here is the summary section:

a. We set up Drumbeat to figure out how to extend our mission beyond Firefox (and beyond software).

b. What we found: Mozilla has an opportunity to build the next generation of web makers. This opportunity is huge.

c. At it’s core, this is about helping makers and creators understand, use and extend the building blocks that make up the web.

d. It’s also about creating a new kind of learning institution and new web tools that invite creativity, tinkering and invention.

e. We can — and should — do these things. They will help us keep the Mozilla spirit alive even as the web changes.

It felt good to look back at these notes. I’m proud of how we’ve focused and refined these ideas. We’ve really doubled down on this original thinking and brought it to life. In particular, I’m proud of where we’ve gone with the idea of ‘new web tools that invite creativity, tinkering and invention.’ Creativity apps for the web could become one of Mozilla’s biggest sweet spots over the coming years: Popcorn and Thimble have given us an early glimpse of this.

It does feel like we left one very important thing out of the bullets above: community and contribution. This really struck me as I re-read my notes. In our early work on the Mozilla Webmaker concept, we did good job of nailing the ‘why’ (create a web literate planet) and the ‘what’  (tools and learning programs the fuel creativity on the web), but we focused much less on the ‘how’ (by working with people around the world who share our vision).

Of course, a great community has sprung up around Mozilla Webmaker. Well over 1,000 of you helped shape our early thinking and ran events as part of our Summer Code Party. But this omission from our early framing does make me wonder: have we put enough emphasis on contribution and community?

My guess is ‘probably not’: we could be doing a better job of finding, supporting and providing value to people around the world who want to help create a web literate planet. Personally, figuring out how to up our game in this area this is my number one priority this fall. I’m going to post more as I dig into this. In the meantime, I’m very much open to suggestions and feedback on this front.

August 19, 2012 06:39 PM

I've uploaded the results from running Linux64 debug builds (both LCOV results and my treemap application). My treemap application has gone through major revisions, too. It now uses CSS transitions instead of the framework's builtin transitions (no more slow script dialogs, and the animations are snappier, although Firefox 14 still chokes on the whole view and Firefox nightly appears to suddenly thrash memory). I've also tweaked the UI a bit to fix minor things that you probably wouldn't notice if I didn't point them out. Instructions on using the view are now in place (it turns out that treemaps aren't as intuitive as I feel them to be). You can also break down results by testsuite, although that feature was very hurriedly hacked in and has very obvious pain points. Code for this portion of the site can be found on github.

The other potentially interesting part is how I get this to work on Try. I don't have this in a public repository yet, but you can follow along the changes on the patch I pushed to try. This is how it works: first, I patch the mozconfigs to include gcov results. Then, I package up all of the notes files and include them in the tarball that contains firefox. Now is where the fun begins: at opportune places in all of the test suites, I output (to standard out) a base64-encoded tarball of all the data files collected during the run of the program. For test suites other than make check, I need to munge the environment to set GCOV_PREFIX to a directory where I can find all of the data files again.

When the magic patch is pushed to try, and after it builds, I can now find in all of the output log files of the build a tarball (or several, in the case of mochitest-other) of all the coverage data. The steps after this are done manually, by dint of only getting the try stuff working last night. I download all of the log files, and extract the tarballs of coverage data into a directory. Then I pull the gcov note files into that directory and run the LCOV tools to collect data. I use ccov (my work-in-progress to replace LCOV) to combine all the test suites into a single file and then produce the data for the nice tree view. LCOV is used to produce the detailed per-file data. After everything is produced, I gather it all up and upload it.

Where to from here? I want to automate the latter part of the process, as babysitting steps that take 10-20 minutes each is surprisingly unproductive. I also want to tackle a replacement for LCOV's HTML output, since it both takes forever to run (well over an hour), and the output is surprisingly lossy in information (you don't know which lines are covered by which tests, and weird corner cases in branch and function coverage could be presented better). Eliminating LCOV altogether is another goal, but I can live with it in the first data collection step for now because gcov does really crazy stuff in coverage. I also want to expand all of these tests to run on more than just one platform—ideally, I want code coverage for all test suites run on all platforms. Assuming, of course, releng doesn't decide to kill me first.

August 16, 2012 11:53 PM

Step 1: Build your version of clang

This should be trivial, although one wrinkle is that you need to specify where the gcc-toolchain is located explictly (/tools/gcc-4.5-0moz3 for now). If you're like me and lazy, you can just use the build-clang.py script to make the final tarball, after tweaking it to include your patch. Note that it expects to be located in specific directories (/builds/slave/moz-toolchain in particular). If you're building by hand, be sure to make a .tar.bz2 of the

Step 2: Place packages in appropriate location

The next script assumes things in particular places. It wants a directory layout that looks like:

$ pwd
/path/ending/in/clang-SVN revision
$ ls
clang-darwin.tar.bz2 clang-linux32.tar.bz2 clang-linux64.tzr.bz2

Step 3: Make manifests

This script is create-manifest.py, which has a requirement of simplejson 2.5 (which is newer than what mozilla-central's virtualenv python provides, alas). If you don't have a new enough python environment, just eliminate the item_sort_key=key_sort parameter and live with the fact that your output files are going to change more lines when importing to mozilla-central. This step produces files like darwin.manifest; these should be copied to browser/config/tooltool-manifests/platform/clang.manifest and the respective releng.manifest. It will also produce files with long, ugly filenames that look like someone dumped out a SHA512 hash as the filename (this is in fact what happens).

Step 4: Upload the files

First, copy the SHA512-named filenames somewhere public, like people.mozilla.org. Next, go into #it on IRC and bug someone about uploading the files to the tooltool server. When they've been uploaded, proceed to the next step.

Step 5: Push to try

Remember to use the modified manifests produced in step 3. On Linux and Linux64, you'll also need to modify the mozconfigs to use clang instead of gcc. An example is here:

export CC="$topsrcdir/clang/bin/clang -fgnu89-inline"
export CXX=$topsrcdir/clang/bin/clang++

Disabling warnings as errors is also probably a good idea, since it seems that Linux people can't stop those extra semicolons sneaking in. Then, push to try and hope for the best!

August 10, 2012 08:48 PM

Gregory Szorc has an interesting post about the execution time of Mozilla makefiles. In it he describes the mechanism that pymake uses to parse a makefile into an AST, and then evaluate the makefile in a specific context. His data shows that the majority of time spent in pymake occurs during the evaluation phase. This is correct, but this slowness is partly a consequence of how GNU make parses and evaluates variables. I believe that it would not be hard to trade minor incompatibility with GNU make and achieve significant speedup in pymake execution.

When GNU make encounters a “normal” recursively-expanded variable, it does not perform any sort of parsing, it merely stores the text. This means that it is possible to invalid syntax into a variable, or to build up a function from multiple appending assignment, for example:

VALUE = aabbcc

TESTVAR = $(subst a,b,
TESTVAR += $(VALUE)
TESTVAR += )

default:
	@echo $(TESTVAR)

The GNU make output of this testcase is “bbbbcc”. Early versions of pymake instead threw a warning about an unterminated variable assignment and the result was ” aabbcc )”.

Building up a makefile variable dynamically may be useful in some special cases, but this is not a technique that most projects use. It does mean that pymake cannot parse variables at the time they are assigned: instead it must wait until the variable is evaluated. (This behavior is in the pymake testsuite).

Variable expansion is by far the most expensive part of core pymake execution. The current parsing method which uses python regular expression engine is relatively slow. Other methods we have tried including hand-parsing using string indexes and were even slower. Parsing all of the config.mk and rules.mk variables as part of the cached AST would significantly improve pymake runtime speed on the Mozilla tree, because we could parse each variable once, instead of re-parsing them at each execution.

The original development versions of pymake did do early parsing, and I only changed it to do parsing at evalation-time when the gmake incompatibility was discovered. I don’t think it would be too hard to switch back. I encourage somebody to try it.

August 08, 2012 04:47 PM

Lucas blogged a while ago about some of the initial work on creating a remote debugging system for Firefox on Android. The desktop patches Lucas mentioned have landed, and remote debugging Firefox for Android is now possible.

Remote debugging requires using Desktop Firefox to connect to Mobile Firefox over USB using ADB. Remote debugging is supported in Firefox 15 and newer releases. Here’s a quick guide to getting started:

1. USB Connection: Connect your Android device running Firefox to a host machine running desktop Firefox using USB. In a terminal, forward the TCP connection using: adb forward tcp:6000 tcp:6000
2. Firefox Desktop: Use about:config to enable remote debugging. Set the “devtools.debugger.remote-enabled” pref to true. Restart. You should see a “Tools > Web Developer > Remote Debugger” menu.
3. Firefox for Android: Use about:config to enable remote debugging. Set the “devtools.debugger.remote-enabled” pref to true. Restart.
4. Firefox Desktop: Choose “Tools > Web Developer > Remote Debugger” and a Remote Connection dialog should open. If you changed none of the other remote debugger preferences, just use the defaults. Press OK.
   
5. Firefox Desktop: You should see an empty debugging window appear.
6. Firefox for Android: You should see a remote debugging connection prompt appear. Press OK to initialize the connection. You have 3 seconds, by default, to accept the connection. You can retry 3 times.
   
7. Firefox Desktop: Once the connection is accepted, the empty debugging window will fill with scripts running in the Firefox for Android web page.
   

Thanks to Lucas, Panos Astithas and the rest of the DevTools team for making this happen. Happy Debugging!

August 06, 2012 07:45 PM

A few months ago, I wrote a post about sane global state via parameters. Now Racket has parameters, but most other languages don't. How important parameters are for a dynamic language was driven home to me while working on getting the Pymake build system working on our tinderboxes.

Pymake is a Python reimplementation of the venerable GNU Make, with two big advantages over it:

1. Parallel GNU Make (at least the MSYS variant) is extremely buggy and prone to deadlocks on Windows, which means the only option is to run serially (-j1). Pymake doesn't have any issues, so build times are much faster on machines with enough cores.
2. The Mozilla build system uses recursive Make. Each recursive call spawns another Make process, and process spawning is rather expensive on Windows. Pymake performs all its recursion within the same process, which speeds build times on Windows even further.

Pymake also provides a few bonus features, one of which is the ability to run Python files "natively", meaning within the same process. This helps speed up builds even more by avoiding spawning extra Python processes. Of course, such scripts have to be written with care to avoid trampling over Pymake itself.

One of the scripts for which this feature was enabled was pythonpath.py. By default, Python only loads modules from the current directory and its system directories. pythonpath.py lets us specify additional directories to load modules from. (Yes, virtualenv is a much better solution, and we're going to switch to it soon.)

To do its work pythonpath.py must modify important global variables like sys.argc and sys.argv. Since it doesn't undo these modifications, it causes hard-to-debug problems down the line. To fix such issues, one would need to consider every eventuality: returns, exceptions, etc. Not only that, one might need to consider modifications by others as well.

As a result, we had to disable this feature for it, slowing down builds by a bit.

If those variables were parameters, one could simply wrap the modifications up in a parameterize and expect things to Just Work. Since everyone would be doing the same thing, things continue to Just Work no matter what is run.

August 04, 2012 08:30 PM

Today we’re releasing Processing.js 1.4.0.  It’s a good solid release, which Jon jokes we’ve been aging for 9 months in order to develop its full flavour.

As we write in the that post, we’re going to change our development process.  In the past we’ve been tracking work in Processing, and chasing performance, parser, and Java-parity bugs.  We don’t plan on doing any major releases for a while, and will instead do minor releases more often going forward.  We still have bugs, and there’s lots of things we could do, but the code has been stable, fast, can compatible for a long time.  And it keeps getting faster as browsers get faster.

The dev team has gotten smaller in the last year, as people have been absorbed into other projects, and as the code has required less attention.  At present it’s Mike Kamermans, Jon Buckley, Yury Delendik, and me, with Mike taking on the leadership role since 1.3.6.  Luckily, quite a few of our current/former devs have been absorbed into the Mozilla Corporation or Foundation as employees and aren’t too far away, including all the names mentioned above plus Scott Downe and Chris Lonnen.  Processing.js has been good for the web, and good for Mozilla, and both make me happy.

Happy sketching!

July 31, 2012 06:38 PM

The side-project I’m working on is coming along nicely, and so I figured it was time to let other people see it. Now, I could just have everyone huddle around my screen, but since many of the people who would be interested aren’t in the same city (or even same timezone) as I am, that wouldn’t work out so well. We tried screen- sharing, but a lot of what’s being worked on is animation, and the frame-rates of the screen-sharing application we were using weren’t up to the task. To get around that, I could have recorded a video, but since a lot of the value of a prototype like this is being able to play around with it, that’s also not a great solution. So, obviously, the best thing to do would be to put it on a publicly available server, and let people run it in their own web browser, whenever they wanted!

Now, I’m running a server or two that I could put it up on, but since the project is related to Mozilla, and since Mozilla offers some personal webspace on one of their servers, I figured I might as well put it up there. :)

So, to make it easy for me to remember to build and upload the code (and to prevent me from trying to figure out all the correct options to rsync every time I wanted to upload the code), I took a couple of minutes to add a command to my volofile, which lets me merely type volo deploy, and have it optimize the code, and copy only the changed files to the remote server.

July 22, 2012 06:04 PM

The testsuites here correspond to running top-level check, mochitest-plain, mochitest-a11y, mochitest-ipcplugins, mochitest-chrome, reftest, crashtest, jstestbrowser, and xpcshell-tests, which should correspond to most of the test suite that Tinderbox runs (excluding Talos and some of the ipc-only-looking things). The LCOV output can break things down by test results, but the treemap still lacks this functionality (I only built in multiple test support to the framework this afternoon while waiting for things to compile).

Caveats of course. This is an x86-64 Linux opt-without-optimizations build. This isn't my laptop, and X forwarding failed, so I had to resort to using Xvfb for the display (which managed to crash during one test run). It seems that some of the mochitests failed due to not having focus, and I have no idea how to make Xvfb give it focus, so not all mochitests ran. Some of the mozapps tests just fail generally because of recursion issues. So this isn't exactly what the tinderboxes run. Oh, and gcov consistently fails to parse jschuff.cpp's coverage data.

Lcov is also getting more painful to use—I finished running tests on Saturday night, and it took me most of Sunday and Monday to actually get the output (!!!). Fortunately, I've reimplemented most of the functionality in my own coverage analysis scripts, so the only parts missing are branch coverage data and the generated HTML index which I want to integrate with my web UI anyways.

July 16, 2012 11:56 PM

As a side-project, I’ve been working on a prototype which is heavily based on a demo page from Stephen Horlander. Now, that page is pretty amazing, but if you look at the source (using command-u or control-u in Firefox, and command-alt-u or control-alt-u in Chrome), you’ll see a lot of code that looks like this:

<div class="menuPanelButton subscribeButton">
  <div class="button"></div>
  <div class="label">Subscribe</div>
</div>

and:

<div class="customizeToolbarItem">
  <div class="customizeToolbarItemIcon share"></div>
  <div class="customizeToolbarItemLabel">Share</div>
</div>

Now, one or two of those would be fine, but when we get into more than that, the repetition really starts to bug me, and I think “Wouldn’t it be better if I could just write stuff like:

<panel-button type="subscribe">Subscribe</panel-button>
…
<toolbar-item type="share">Share</toolbar-item>

instead?” And it turns out I can, using a new library called x-tag! The first thing I need to do is register the new tags I’ll be using. That’s done with code like this:

// These first two lines are here because I’m using require.js, which I’ll
// talk about in a future blog post…
define(function (require) {
  require(["jquery", "x-tag"], function($, xtag) {

    // And this is the meat of the functionality.
    // First, we’ll register the new "panel-button" tag.
    xtag.register("panel-button", {
      onCreate: function(){
        var self = $(this);
        // When the tag is first seen, make the innerHTML be this stuff below.
        self.html("<div class='menuPanelButton " + self.attr("type") + "'>" +
                    "<img src='images/button-" + self.attr("type") + ".png'" +
                    "     class='button'>" +
                    "<div class='label'>" + self.text() + "</div>" + 
                  "</div>");
      },
    });

    // And then, we’ll register the new "toolbar-item" tag.
    xtag.register("toolbar-item", {
      onCreate: function(){
        var self = $(this);
        // We could also replace this element with the html below, but I
        // haven’t done that here because I haven’t needed to yet.
        self.html("<div class='customizeToolbarItem'>" +
                    "<div class='customizeToolbarItemIcon " + (self.attr("type") || "") + "'></div>" +
                    "<div class='customizeToolbarItemLabel'>" + self.text() + "</div>" +
                  "</div>");
      },
    });

  });
});

The second step is to replace all the old html with the new tags. (I did that, too, of course.) And there we go. That’s it. In the file I was modifying, the combination of that and moving the javascript out into a separate file took the html from 275 lines down to 146 lines, and let me more easily change the buttons around, and add new ones. I call that a win, and from now on, whenever I see large blocks of repeated html, I’m going to be seriously tempted to switch them to an x-tag!

One caveat I will mention is that in my first attempt, I tried to use both the content property, and the onCreate method, and that totally didn’t work, since the content would be replaced by the value of the content property long before I had a chance to muck around with it in the onCreate. So in the future, I think I’ll just jump straight into using the onCreate method, since it’s not that much harder.

July 13, 2012 01:39 AM

LDAP

Ah, LDAP. The big red part of comm-central whenever I make my coverage treemaps. The problem here could be solved if we had an LDAP fakeserver; having written both IMAP and NNTP servers, this shouldn't be hard? Except that LDAP is not built off of a textual-based layer that you can emulate with telnet but an over-engineered protocol called ASN.1 and more specifically one of its binary encodings. The underlying fakeserver technology is built with the assumption that I'm dealing with a CRLF-based protocol, but it turns out that, with some of my patches, it's actually easy to just pass through the binary data (yay for layering).

The full LDAP specification is actually quite complicated and relies on a lot of pieces, but the underlying model for an LDAP fakeserver could rather easily be controlled by just an LDIF file with perhaps a simplified schema model. At the very least, it's a usable start, and considering that the IMAP fakeserver still isn't RFC 3501-compliant 4 years later, it's good enough for testing.

Here, a big issue arises: the actual protocol decoding. I started by looking for a nice library I could use for ASN.1 decoding so I don't have to do it myself. I first played with using the LDAP lber routines myself via ctypes, but I found myself dissatisfied with how much work it took just to parse the login of the LDAP serve. I then looked into NSS's structured ASN.1 decoding, even happening upon a nice set of templates for LDAP so I didn't have to try to build them with the lack of documentation, but it still ended up not working well, especially given the nice model of genericity I was looking for. I played around with a node-based LDAP server (especially annoying given the current name feud in Debian that prevents the nodejs package from migrating to testing). It worked well enough for an initial test, but the problem of either driving the server from xpcshell or writing node shims combined with the fact that it only processes the protocol and has no usable backend caused me to give up that path. Desperate, I even tried to find just general BER-parsing libraries in JS on the general web and discovered that the ones that were there couldn't quite cope with the format as we use it.

Conclusions: it's possible. The only real hard part is writing the BER parsing library myself. If anyone decides they want to work on this, I can send them the partial pieces of the puzzle to finish. If not, I'll probably nibble on this here and there over the next year or two.

MIME

MIME—that's well-tested per our testsuite, right? Well, not really. A lot of the testing is just pure incidental: hooking the MIME library up to the IMAP fakeserver did a good job of fleshing out a lot of issues, but you can also find lots of small details that no one's going to notice (charsets come to mind). It turns out that MIME is one of those protocols where everybody does the same thing slightly differently, and you end up accumulating a lot of random fixes to MIME. If you want to replace the module from scratch, you become terrified of finding random regressions in real-world mail.

Perhaps unsurprisingly, there are no test suites for proper MIME parsing on the web. There is one for RFC 2231 decoding (kind of). But there's nothing that tries to determine any of the following:

• Charset detection, especially who gets priority when everyone conficts
• Whether a part is inline, attached, or not shown at all
• How attachments get detected and handled
• Test suites for the various crap that crops up when people fail at i18n
• Text-to-html or HTML sanitization issues
• Identifying headers properly (malformed References headers, etc.)
• Pseudo-MIME constructs, like TNEF, uuencode, BinHex, or yEnc
• S/MIME or PGP

Issues relating to message display could be handled with a suite of reftests. A brief test confirms that reftest specifications accept absolute URLs, including the URLs that are used to drive the message UI (this can even test it from loading the offline protocol). Reftests even allow you to set prefs before specific tests; with a bit of sugaring around the reftest list, a MIME reftest is easily doable. Attachment and header handling could also follow a MIME reftest design, but I'm not sure that is the best design. I'd also like it to be the kind of test that other people who write MIME libraries could use.

The main issue here is seeding the repository with something useful. Sampling a variety of Usenet newsgroups (especially foreign-language hierarchies) should pick up something useful for basic charset, and I can get uuencode and yEnc by trawling through some binary newsgroups. For a focus on gmail, I could probably pick up some Google Groups things (especially if I recall the magic incantations that let me at actual RFC-822 objects). Random public mailing lists might find something useful. My own private email is unlikely to provide any useful test cases, since I tend to communicate with too homogeneous an environment (i.e., I don't get enough people using Outlook). Sanitizing all of this public stuff is also going to be a pain, especially with the emails that have DKIM.

OS integration

OS integration is a nice header for everything that involves the actual OS: MAPI, import from standard system mail clients, integration with system address books. Unfortunately, my main development environment is Linux, where we have none of this stuff, so I can't really claim that I have a plan for testing here. Thanks to bug 731877, at least testing Outlook Express importing is a possibility, but true tests would probably require dumping some .psts into our tree, but we have no similar story for Mail.app. MAPI could be done with a mock app that exercises the MAPI interfaces; what it really comes down to is that we need to implement these APIs in a way that we can test them by executing in various mock environments during tests.

Performance tests

The other major hole we have is performance. Firefox measures its performance with things like Talos; Thunderbird ought to have a similar testsuite of performance benchmarks. What kind of benchmarks are useful for Thunderbird testing though? Modulo debates over where exactly to place the endpoints on the following tests, I think the following is a good list:

• Startup and shutdown time
• Time to open a "large" folder (maybe requiring rebuild?) and mem usage in doing so
• Doing message operations (mark as read, delete, move, copy, etc.) on several messages in a "large" folder. Possibly memory too
• Time to select and display a "large" message (inline parts), as well as detach/delete attachments on said message
• Cross-folder message search (with/without gloda?)
• Some sort of database resync operation
• Address book queries

For the large folders, I think having a good distribution of the size of threads (so some messages not in threads, others collected in a 50+ message thread) is necessary. Slow performance in extra-large folders is something we routinely get criticized on, so being able to track regressions is something that I think is useful. Tests that can also adequately catch some stupid things like "download a message fifteen times to display it" are extremely useful in my opinion, and I feel like there needs to be some sort of performance tests that highlight problems in IMAP code would be useful.

July 11, 2012 03:55 AM

I was recently asked to provide a list of all the plugin-related changes from Firefox 11 to 12 to 13. This is actually not the easiest question to answer: both bugzilla and source code searches may produce false negatives and false positives.

Using bugzilla to answer this kind of query fails whenever the bugzilla metadata is incorrect or misleading. Usually the Target Milestone for a fixed bug indicates the release version that it first ships in, but it is not uncommon to forget to set the TM for bugs. And when a bug is backported to an earlier train, the target milestone doesn’t always get moved backwards to match. In addition, filtering by bugzilla component often leaves out important bugs, since changes to graphics, DOM, layout, or JS can make noticeable changes in plugin code.

Instead, I decided to use Mercurial to answer this question. Almost all changes which affect plugin function are going to be located in the following directories and files:

• dom/plugins
• content/base/src/nsObjectLoadingContent.{h,cpp}

So what I really wanted was a list of changes between two Firefox releases which touch that set of files. But how can I use Mercurial to search only the revisions between two specific releases? Here is the branch diagram for the Firefox trains:

By default, hg log -rfoo:bar will list the changes between two particular revisions in whatever linear order these revisions happened to be pulled into your repository. This means that you may get different results depending on the order you pulled. But there is a feature in Mercurial called “revsets” that lets you pick changes in much more specific ways. For more information about revsets, issue hg help revsets or see the online documentation.

In this case, I wanted to log all the revisions between the mozilla-central branch point of Firefox 12 and the release of Firefox 13. Revsets can figure the branch point with the ancestor operator, and can then list all the changesets between two revisions (including all branch/merge revisions) with the :: operator. So the final command for finding all changes which touch plugin files between Firefox 12 and 13 was:

$ hg log -r "ancestor(FIREFOX_12_0_RELEASE,FIREFOX_13_0_RELEASE)::FIREFOX_13_0_RELEASE" dom/plugins/ content/base/src/nsObjectLoadingContent.{h,cpp}

Special thanks to Dirkjan Ochtman (djc) for introducing me to revsets!

July 09, 2012 03:38 PM

To say that the recent announcement on Thunderbird's future threw people in a tizzy would be an understatement. After all, we have nothing less than apocalyptic proclamations of the death of Thunderbird. I believe that such proclamations are as exaggerated as Samuel Clemens's death notices (apologies for making a joke that is probably inscrutable to non-en-US people).

The truth is, Thunderbird has not been a priority for Mozilla since before I started working on it. There really isn't any coordination in mozilla-central to make sure that any planned "featurectomies" don't impact Thunderbird—we typically get the same notice that add-on authors get, despite being arguably the largest binary user of the codebase outside of mozilla-central. Given also that the Fennec and B2G codebases were subsequently merged into mozilla-central (one of the arguments I heard about the Fennec merge was that "it's too difficult to maintain the project outside of mozilla-central") and that comm-central remains separate, it should be quickly clear how much apathy for Thunderbird existed prior to this announcement.

As a consequence, the community has historically played a major role in the upkeep of Thunderbird. The massive de-RDF project was driven by a lawyer-in-training. I myself have made significant changes to the address book, NNTP, testing, and MIME codes. Our QA efforts are driven in large part by a non-paid contributor. More than half of the top-ten contributors are non-employees, according to hg churn. So the end of purely-Thunderbird-focused paid developers is by no means the end of the project.

There's a lot of invective about the decision, so let me attempt to rationalize why it was made. Mozilla's primary goal is to promote the Open Web, which means in large part, that Mozilla needs to ensure that it remains relevant in markets to prevent the creation of walled gardens. I believe that Mozilla has judged that it needs to focus on the mobile market, which is where the walled gardens are starting to crop up again. In the desktop world, Mozilla has a strong browser and a strong email client, and maintaining that position is good enough. In the mobile world, Mozilla has virtually no presence right now. Hence all of the effort being put into Firefox Mobile and B2G right now.

Now, many of the decisions as to the future of the project are uncertain; unfortunately, the email laying all of this out was prematurely leaked. But it is clear that Thunderbird suffers from massive technical debt: when I was pondering parts that the Gaia email app might be able to leverage, I first considered the IMAP protocol implementation and then ran out of things to suggest. Well, maybe lightning or the chat backends (for calendaring and IM, respectively), but it's clear that most of the mammoth codebase is completely unsuitable for reincorporation into another project. To this end, I think the most useful thing that could happen in Thunderbird falls under the "maintenance" banner anyways: a replacement of these crappy components with more solid implementations that are less reliant on maybe-obsolete Gecko features and that could be shared with the Gaia email app. As a bit of a shameless plug, I have been working on a replacement MIME parser with an explicit eye towards letting Gaia's app use it. Such work would be more useful than whining about the decision, in any case.

July 09, 2012 04:14 AM

Yesterday afternoon, I watched a video from James Long about Mortar, which is a template for making HTML 5 Open Web Apps. Now, coincidentally, I’m starting a new project (in my spare time, obviously, since it’s a Sunday), and while it’s not an Open Web App, I saw no reason not to use the same tools they were using.

Of course, since nothing’s easy, I ran into a problem pretty quickly. My problem was that every time I tried to require("jquery-ui");, I got an error of “ReferenceError: jQuery is not defined”. There wasn’t a lot of information about how to fix it, so after most of an afternoon mucking around, I finally came up with something that seems to work, and thought I would post it.

1. Go into your www/js/lib directory.
2. curl -O https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.18/jquery-ui.js
3. Edit the jquery-ui.js file. At the top add the line define(["jquery"], function (jQuery) {, and at the bottom, add the line });.
4. That’s it. From there you should be good to go!

Now, I suspect there’s a better way to do this, and hopefully James or Bryan will jump in the comments and tell me what it is, but for now, at least this works.

July 08, 2012 10:14 PM

tl;dr — I love Twitter and I love git, and wouldn’t it be nice if I could build the former on the latter?  What follows is a flight of fancy for your Friday feed reading…

Introduction

I went on a bike trip with my family last weekend, and when I came home, everyone was in flap about how Twitter was going to screw third-party developers.  Numerous people started the familiar chant about how Twitter is a core infrastructure of the web, that a for-profit company driving it is going to end badly, etc.  I have to say that I agree with all of this.  Twitter seems too important to me to have its direction come from the need to make money.

How do you recreate Twitter?  This isn’t an “I could build Twitter in a weekend” post, since I have too much respect for the kinds of problems they are solving, the scale involved, etc.  Instead, this is a thought experiment.

If you’re a developer, you’re probably already using git and github.  If you don’t use them, think of git as a database, and github as free hosting for your database.  There are lots of free git hosting options, but github has captured the hearts and minds of the developer community.  Github lets me, you, and your dog (your dog needs a Twitter account too, right?) host open source projects for free, and gives us 300M of storage to do so.

So, what is a tweet?  Let’s simplify and say it’s max 140 characters, written by a particular person, possibly in response to another tweet, at a particular time, potentially containing links, and maybe links to Twitter objects.  And what is a git commit?  A (possibly empty) set of file changes, with an author (and potentially different committer, which will be important below), a commit message, a date, and a globally unique identifier sha (e.g., 78dc050439110a2949f1d32df2c69962dcf92258).  A commit message is typically meant to be human readable text, and a commit to contain files; but none of those has to be true.  With a bit of cutting, some JSON, and imagination, the two are compatible.

My basic idea is to use git clones/forks, hack commit messages, and store metadata in files (details in a minute). More importantly, the aim of gitter would be to enable the ease of social networking without the requirement that one give up his or her control over content. I would own my gitter content, and manage it.  By using a distributed architecture, and one which already scales and has redundant backup via parallel scaled infrastructure, gitter hopes to reduce the need for centralization. Gitter would also allow for open licensing of content, and more, personal control over licensing. Gitter eliminates unwanted ads, spam, etc. by putting each individual in control of the users they follow.

Gitter

Gitter uses git as a distributed filesystem, and github.com as primary git repo host (note: other git hosts are possible without affecting the architecture). Every gitter user has a fork/clone of the gitter repository.  All gitter-messages are stored as JSON strings in the log message of an empty commit. All gitter-message commits are done to a “stream” branch in sequence. As a result, every gitter-message is globally unique, and identifiable via a git commit sha, and includes metadata such as author, date, etc.

It is important to emphasize that git and github are used as a backend, but in no way do I assume users will interact with either directly (though you could). Instead, operations on git and github will be done via clients. Using git and github gives more than 1 million users (I suspect it’s much higher now) instant access to the gitter messaging system without creating a new account, and free, open accounts for anyone not already a user.

Users

Users of gitter begin by forking the gitter repo. This gives them a repo with two branches: master and stream. The master branch contains a number of files.

The first file contained within each gitter repo’s master branch is profile.json. The profile.json file contains information about the user, including things like:

{
  "name": {
    "first": "David",
    "last": "Humphrey
  },
  "login": "humphd",
  "nickname": "humph",
  "twitter": "humphd",
  "license": "http://creativecommons.org/licenses/by-nc/3.0/",
  "email": "david.humphrey@email.com",
  "urls": [
    "http://vocamus.net/dave"
  ],
  "locale": "en-CA",
  "timeZone": "GMT-05:00",
  "bio": "Bio for David Humphrey"
}

The profile.json file describes a gitter user, and indicates important hints for clients, for example the default license for all messages written/published by this user.

The second file contained within each gitter repo is following.json. This file contains the list of other gitter users, and their repos, that the current user follows.

The gitter repo also includes an orphaned, empty branch called stream. This is the branch where gitter messages go–more details follow.

Following Users

Every gitter user has a fork of the gitter repo. As a result, every user has a unique URL, for example:

git://github.com/someone/gitter.git

Following a user means doing two things:

1) adding their forked repo as a read-only remote to your gitter fork:

git remote add someone git://github.com/someone/gitter.git

2) adding the user’s repo information to following.json in the master branch

Here the user at git://github.com/someone/gitter.git was added using their default github username. However, the only thing that matters globally is the URL–we can use whatever name we want. Unlike Twitter, G+, Facebook, etc., where two people can’t share the same name, in gitter you can refer to people using any common name you want. If I know git://github.com/someone/gitter.git as dave, I can use that shortname. In fact, users can provide a preferred nickname in their profile.json file as a clue to clients. In other words, we can:

git remote add [nickname] git://github.com/someone/gitter.git

This means that two users can follow the same person (i.e., follow git://github.com/someone/gitter.git) and choose to refer to him/her with different names. The gitter client will make sure there is no confusion internally, by always using URLs instead of short names.

The remotes allow git to manage internal links to other users’ repos, but don’t provide visibility to other users–there is no way to find out who you follow. Therefore, we also keep a list of other gitter users we follow in the following.json file. The file looks like this:

[
  {
    "user": "jbuck",
    "nickname": "jon",
    "twitter": "jbuckca",
    "url": "git://github.com/jbuck/gitter.git"
  },
   {
    "user": "someone2",
    "twitter": "humphd",
    "url": "git://github.com/someone2/gitter.git"
  }
]

The following.json file is an array of objects describing the users this user follows. Each user object contains a user field, which is the github username for this user. Next comes an optional nickname, which allows us to specify a different name for this user than the one they use on github. Next comes an optional twitter username, in case one wants to map or cross-post messages from gitter to Twitter. Finally, a url for their gitter repo fork.

Unfollowing means removing a user’s remote from the forked gitter repo of a user, as well as from following.json on the master branch.

Messages

A message is an escaped sequence of text. No HTML is allowed. Messages may be any length, but 140 characters is encouraged for compatibility with Twitter. Messages are stored in JSON strings. Here is the most basic example:

{
  "text": "Writing a message in gitter!"
}

Publishing this message involves doing an empty commit on the stream branch of the user’s fork of the gitter repo:

git checkout stream && git commit --allow-empty
  --message='{"text":"Writing a message in gitter!"}'

And then pushing:

git push origin stream

And in github you get:

A message can be simple text. A message can also contain various types of data, and refer to external data.

A message might include a URL, which is itself simple text:

{
  "text": "Writing a message in gitter that
           includes a link to http://gitter.org"
}

A message might include references to external gitter objects, for example, another user:

{
  "text": "Writing a message in gitter that refers to
           @jon and @humph.",
  "users": {
    "jon": "git://github.com/jbuckca/gitter.git",
    "humph": "git://github.com/humphd/gitter.git"
  }
}

In this example, two gitter users are referenced in the text. The author has used the nicknames @jon and @humph, which in turn refer to URLs of remote gitter repo forks. Other clients displaying this message might choose to show the nicknames differently, for example, if another user knows Jon as jbuck instead of jon. Internally the mapping will be back to the URLs. By cross-referencing the following.json file for this user, other information can be obtained about each user, for example, a Twitter username.

A message might include a reference to another gitter message. This can happen when a user Replies to another message, References something someone else said, or does the equivalent of a Retweet (RT):

{
  "text": "Writing a message in gitter that refers to
           another message @jon wrote",
  "users": {
    "jon": "git://github.com/jbuckca/gitter.git",
  },
  "references": [
    "git://github.com/jbuckca/gitter/commits/ec52b6907a1988f042eb61ac3450cf8361ef60b4",
  ]
}

Here I can track back the author of a message referenced in the current message. The fact that messages can reference one another removes the necessity that I copy things into the text itself–the client can surface this information alongside the message. In this case the original message can be displayed alongside the current message, in order to preserve conversational context.

The concept of a Retweet in Twitter can be done using git’s ability to cherry-pick commits from one branch to another. If I want to re-broadcast a message from one of the users I follow to the users who follow me, I only need to cherry-pick the commit from their remotes//stream branch onto my stream branch. Git will keep the author and committer information, thus making it possible for the original author’s info to get recorded in my message stream.

All messages are published under the user’s desired license. This is either the default gitter license (TBD), the license specified in the user’s profile.json file, or the license specified in the message itself:

{
  "text": "This is a message for which I want
           to specifically use license XYZ",
  "licenses": [
    "http://licenses/xyz"
  ]
}

In this case the license(s) for this message overrides any other implied license for this user or gitter in general. It is up to the client to surface this information.

Updates

As users write and publish messages, their followers keep up-to-date with them via

git fetch --all

This pulls in all commits, and thus messages/payloads, from the users (i.e., remotes) this user follows. Every user’s messages exists via the remotes//master branch pointer. Accessing and using the messages on these branches is possible without ever merging.

Other ideas

It would be wonderful to leverage Github’s gh-pages branches in order to automatically publish a web-reachable version of a user’s stream. In this way, everyone would have a standard URL where I could go and look at their messages, http://humphd.github.com/gitter/ is akin to https://twitter.com/#!/humphd. Providing a “Follow Me” link on this page, a way to see who this person follows, etc. would also be nice.

What about publishing things referred to in messages? If I want to share an image or a PDF, and don’t have a place to host it? Because all messages are associated with commits, each message can also include a file or files.

{
  "text": "This is a message which has a PDF payload",
  "payload": [
    "git://github.com/jbuckca/gitter/commits/ec52b6907a1988f042eb61ac3450cf8361ef60b4/document.pdf"
  ]
}

Here the user has chosen to publish the PDF document to which they refer in the message. This document, document.pdf, is committed along with their message. Followers of this user can choose to obtain the document remotely (i.e., via payload URL), or can save it locally (e.g., git fetch). In this way distributed publishing of content larger than appropriate for a message can be done easily. Another way to one-click publish media to the web would be to use Github’s Downloads page: http://developer.github.com/v3/repos/downloads/

Problems

Of course there are all kinds of problems I’m skipping over. Let’s consider some of them.

How do we support conversations among people who don’t follow each other? I often have people who I don’t follow reply to things I write on Twitter. It’s a great aspect of the platform, and one I don’t have a good solution for above.

How do we deal with Github’s rate limiting? You only have 5000 API calls per hour. I wrote a node.js client for what I did above, but it’s unrealistic to go this route without having the ability to do more calls.

If we don’t use the Github API, and do raw git repo commands, how do you do this on mobile? It’s probably not realistic to have a git repo on your mobile phone due to size. Maybe that’s only an issue today, and won’t be in a few years.

How do we manage state between browsers/devices? That is, if I’ve read your tweet at 3:00pm, and I move to my mobile phone, how do I skip pulling that one in? I’d need to record the commit sha of the place I left off, but where to put that? I could put it in the following.json file, but that means I have to push a ton of commits to my repo every time I get new messages.

This also raises the problem of bandwidth. I currently follow 265 people on Twitter. In order to get updates, I need to pull in commits from 265 repos, or do 265 calls to the Github API. And I need to do that every N minutes (1, 2, 3?). That’s potentially a lot of traffic. How to reduce it?

Conclusion

I’ve actually written some code to experiment with the above, and it’s not really workable yet. However, I’d love to see it get to a state where it is workable. There should be no reason why I can’t have complete history of my messages and those I follow, why I shouldn’t be able to specify a license for what I write, and I shouldn’t be dependent on a single company to control my communication.

I’m looking forward to using gitter some day soon.

Thanks to Jon Buckley for helping me think through some of the details above.

July 06, 2012 06:02 PM

We want everyone to tap into the full creative power of the web. That’s the point of Mozilla Webmaker. Part of  this is about people: building a global community of webmakers. But another essential element is building tools that both invite people to make cool things on the web and that help them learn how the web works. Last week, we released early versions of two of these tools: Thimble and Popcorn. This post offers background on these tools plus musings on next steps.

A month or two back, Free Software Foundation Chief Counsel Eben Moglen said: “We made the web easy to read, but we didn’t make the web easy to write. Facebook took advantage of this gap.” This is a useful way to look at the challenges the web now faces.

Over a billion people are now on social networks. They use these networks to create and share (awesome!). But they do so only on the terms social networks offer them. Most people have neither the tools nor the skills to tap into the unbounded creative potential of the web (e.g. I want to change how this app works, let me hack it). This seems like something worth fixing.

The goal of Mozilla Webmaker is exactly this: to move people from being users of the web to being makers of the web. While part of this move is about new skills and attitudes, another part has to be about tools and content. Specifically: tools and content that make it easier to create using the full power of the web. Easier to customize a blog template. Easier to add a data layer to a video. Easier to hack a game. And so on. As Mozilla begins to build tools like this, I see three big buckets of things we need to get done:

1. Build a foundation: Thimble + Popcorn as way to test our making + learning thesis (2012)
2. Build with the community: add in badges, get community adding content and code (2012+)
3. Make the app world hackable: add in JavaScript, game hacking, app hacking (2013+)

Of course, this isn’t just about Mozilla: we’re already working with Tumblr, Codecademy and many others who are also building tools that encourage making and learning. But Mozilla does have a clear role to play here, especially around tools that build in the design principle of ‘making as learning’ from the beginning. This is our focus.

Step 1 – Build a foundation

Quickly ship tools that test our webmaker thinking: this was our plan with Thimble and Popcorn. Earlier this year, we agreed we needed to try out our ‘people learn by making’ thesis soon and in the real world.

With this in mind, we designed very simple tools with a collection of ‘hackable’ projects at the centre. Each project is a web page or interactive video template that gives people a) a starting point for making something and b) instructions that help you learn while making.

In Thimble, for example, each project is a single, simple HTML page. All pages include code comments that suggest what part of the page you might hack and what techniques to use. Eg. <!– This is a comment. These comments tell you what to do. The next section is a <H1> header. Edit the text within the header tags to change what words are on the web page. –> Thimble currently includes about a dozen making + learning templates that teach basic HTML and CSS while letting you make a meme, kill zombies or remix an animal from the London Zoo.

In Popcorn, the focus is much more on learning the mechanics of the web while also learning how to tell stories in new ways. The core element of a Popcorn project is a video wrapped in an HTML page template. You are invited to tell your own story by manipulating the video and the template. For example, with the Robots template, you design your own Robot invasion movie by adding in Google maps (learn how APIs work + target the robots at your home town), by changing the words the robots say (learn about variables and speech synthesis), or by adding in pictures of people and places the robots will target (learn about links and embedding). Similar templates exist where you can make your own web newscast or create a VH1-style pop up layer on top of your video.

Popcorn and Thimble share a common design philosophy. They start from the premise that remix and making are central to how people have always learned the web. The web is an open, view source world where it’s easy to see a technique in action and then copy it. We’re trying to make that even easier and more explicit by offering tools and pre-made projects that help people learn by making. We think this approach is especially promising for the 1 billion+ people on social media who clearly like to express themselves on online but have no plans of becoming a capital P ‘programmer’. For these people, the starter content we’re offering will be much more important than the tools themselves.

Our goal for this 1st step in the Mozilla Webmaker tool roll out: test this making + learning thesis. See if people like what we’ve made so far. Evaluate whether they are learning anything and what they are learning (see ‘badges’, below). We’ll do this testing and evaluating over the next six months while also shipping improvements plus new content for Thimble and Popcorn.

Step 2 – Build with community

Our next big priority needs to be building Mozilla Webmaker with a community. There are two parts to this: a) building basic tools that offer value to people who want to make, learn or teach and b) creating an architecture of participation that makes it easy for people to contribute content and code. We’ve already started on a) (see above), now we need to get moving on b).

We’ve got a few early wins. Some of the best Thimble projects came from the hack jam we held with NESTA in London about a month ago. Popcorn already has a base of open source code contributors. And, of course, we’ve got well over 500 people who have stepped up to organize a #mozparty this summer. These are big contributions already.

What we don’t yet have is a systematic way for more people to get involved, especially on content, code and localization. As an example: we want to get to a place where large numbers of people contribute Thimble and Popcorn projects for other people to make / learn on top of. I believe that this sort of community-made content will be key to the growth of Mozilla Webmaker. Unfortunately, we don’t have a good way for people to do this yet.

As a first step towards fixing this, we’re going to create an easy way for people to propose Webmaker.org and Thimble content simply by posting it to a Mozilla wiki. We’re working on the specifics right now, but the ideas is that people can review / refine / discuss stuff there, and then we can port the best stuff to our main sites when it’s ready. Obviously, we need something easier and more robust over the longer term: a gallery or repository where anyone can post content ideas into the Mozilla Webmaker universe. We also need easier ways into the code side of our projects. These are both things we’re thinking through with the aim of making progress this year. We’re very open to your ideas.

We also need a way to see who’s who in the community: to know what people are working on; to find people with particular skills; to find people with similar interests. This starts with simple communications tools like the new webmaker mailing list and the #mozparty Twitter hashtag. But we need something more robust and something that works across alot of platforms if we want Mozilla Webmaker to scale.

This is where Mozilla Webmaker badges come in. By the end of 2012, we plan to offer badges to recognize the skills that people have learned and that recognize contributions to the community. The skill badges will include things like “I understand HTML basics” and “I’ve helped 5 friends learn CSS”. We’re still working out the specifics. The contribution badges will recognize those who have helped others and those who have contributed learning content and code. This will give people within the Webmaker community an easy way to signal each other — to know who knows what and who is into what. More importantly, it will also give people a way to show what they learned and for us to monitor what and whether people are learning.

We’re starting work on this 2nd step right now, building a more systematic architecture of participation in parallel with our efforts to test and improve our first set of tools.

Step 3 – Make the app world hackable

Our third big step for Mozilla Webmaker tools has to be ‘teaching real programming‘ – getting people to the point where they can create or remix a game, app, etc. We chose to start Mozilla Webmaker with a focus on HTML and video so we could test our making and learning thesis using basic content that almost anyone on the web can make. But, our ultimate goal is to let people control, configure and create all corners of their digital lives. Games. Apps. Social networks. Civic participation sites. Science projects. And so on. At some point, that means teaching programming, or making programming easier to do.

It’s in this last phrase where one of our big questions sits: do we teach ‘real code’ like JavaScript, or do we adopt / develop a simple tool for producing JavaScript etc. like Meemoo or Blockly?

Our plan for 2012 is to explore both paths. We’re working with Codecademy to get their content on Webmaker.org and to encourage people to use their courses as part of #mozparty. This gives people a ‘learn real code in the abstract’ option. We’ll likely do the same with Meemoo, Blockly and other promising tools: write them up on Webmaker.org; encourage people to use them; and then ask people to feedback on what they observed. This is a good way to promote the work of allies who also want to teach the world to code while at the same time investigating whether there are gaps a tool like Thimble++ could play.

There are already a couple of places where I think Mozilla-made tools — and Mozilla’s learning philosophy — could add value in the ‘learn programming space’. One opportunity is in the realm of hackable games: HTML5 games which are designed from the ground up to be modified. People could be invited to change the game mechanics and or to bring in content from across the web. E.g. imagine grabbing your friend’s picture from Facebook and mapping it onto a rock in Angry Birds. Another opportunity is in mobile apps, especially those designed for rich on-the-fly HTML5 content creation. Eg. imagine a Popcorn-enabled Boot-to-Gecko phone that pulled in all kinds of contextual content and data on the fly, feeding a ready to edit package of content back to your laptop via the web. We’ll explore ideas like these this year at a blue sky and maybe even prototyping level to see what’s possible.

My guess is that this 3rd step in the Mozilla Webmaker tool effort won’t really move into high gear until early 2013. There are a number of experiments planned for later in 2012, but these are mainly about seeing what’s possible and giving us enough insight that we can develop a solid roadmap for our work in this space.

–

Admittedly, this is a pretty big dream. Mozilla Webmaker is ambitious on purpose. We need this kind of ambition if we want a world where we all understand and can shape our digital lives.

But we also need a plan to turn ambition into reality. We made the beginnings of a plan when we launched Mozilla Webmaker earlier this year (you can read it here). I’ll post a review of where we’re at with this plan sometime in later July, including how what we’ve learned from tools like Thimble and Popcorn is helping us evolve the plan.

In the meantime, there is one thing we know for sure: Mozilla is way too small to take on the Webmaker vision on its own. As I said above, we need to a) figure out how we can provide something valuable to others who want to teach the world to code and b) get at least some of these people working with us directly to create the tools, content and community for webmakers. Finding ways to work with people who share our vision (is this you?) is the number one priority of the Mozilla Webmaker team right now.

Which leads me to two closing asks: let us know if any of the work we’re doing on Mozilla Webmaker tools can help you + let us know if you want to help. It’s not always easy to find your way in. We know that and can help. The best place to start is on the Webmaker mailing list or our weekly community conference call. You can also just post a comment here or send me email. Just show up, put up your hand and say ‘I want help / how can I help?’ Someone on the Mozilla Webmaker team will be there to help you figure it out from there.

July 02, 2012 04:07 PM

Josh Matthews' amazing BugsAhoy now has support for the build system. All open bugs labelled [good first bug] and with a listed mentor in the Core::Build Config and MailNews Core::Build Config components are listed. If you're interested in one of the bugs, post a comment there or find us in #pymake on irc.mozilla.org.

June 30, 2012 06:25 PM

Ingredients:

• 1 (one) copy of clang, preferably on the new side
• 1 (one) source tree containing mozilla-central
• 4 (four) GiB (that's 2³⁰, not 10⁹) of memory
• Linux (these instructions not tested on OS X, so it may or may not work for you)

Instructions:

For the sake of example, I have clang installed to /usr/local; substitute as appropriate for your environment. Configure mozilla-central with the following mozconfig:

# -O4 tells clang to do LTO
ac_add_options --disable-debug --enable-optimize='-O4'
# Exclude this line, and your will use over 10x the amount of memory.
ac_add_options --disable-debug-symbols
# Build with clang
export CC=/usr/local/bin/clang
export CXX=/usr/local/bin/clang++
# These flags are necessary to get elfhack to compile
export LDFLAGS='-Wl,-z,norelro -flto'
# This flag is necessary to link properly
export RANLIB='ar -s --plugin /usr/local/lib/LLVMgold.so'

You may season with other configure options as desired. Configure and let build for a few hours. Admire the build process as most individual files get compiled much quicker, but be displeased when link times explode. Enjoy!

June 27, 2012 07:21 PM

My thinking often returns to tools.  Whether philosophically or practically, the idea of the tool fascinates me.  I first became aware of the question of tools through the writings of philosophers like Plato, Aristotle, Heidegger, and Gadamer–tools, devices, equipment, the in-order-to, the ready-to-hand, the hammer vs. the hammering-hand.  One of the difficulties of tools is that they rarely reveal themselves, unless they are broken.  We don’t think about the hammer when it is hammer, and we are hammering; we might notice it if we hit our thumb or bruise the wood.  Thinking about tools requires one to put them down long enough for them to become visible as objects separate from the intention of their users.

Most writers, when they speak of tools, think in terms of the physical world.  I come from a long line of farmers and tinkerers for whom tools in the workshop, barn, or basement were a critical component of one’s existence.  My father has an amazing array of tools, many of which I have no idea how to use.  And yet I too have an enormous collection of tools.  The tools I use most often are made of software, and it is toward these tools that I now want to turn.

I had an interesting conversation with Christian Legnitto a few weeks ago on the subject of tools in software (one, two, three, four, five also, Twitter, why is it so hard to link to this?).  That conversation foregrounded tools once again, and at Johnathan’s request, I’m going to try and say something about the place and purpose of tools in software projects.

The first thing to notice about tools in software, what I’ll call tools() in order to differentiate them from what we normally mean when we say tools, is that they don’t exist in a form separate from the thing on which they operate.  Any trip into my father’s basement or garage instantly reveals the presence of certain objects.  These tools have a physicality beyond their intentionality and purpose.  The process of the hammering-hand and the physicality of the hammer hanging on the wall are distinct.

On the other hand, when one examines a tool(), it becomes clear that there is only process.  A tool() is a repeatable process run at speed and scale.  A tool() is what I used to do manually but have now automated.  One of the main goals of a tool() in a software project is to provide the means for an again.  The tool() isn’t something I pickup or use so much as something I’ve done, and am intending to do again.

What do tools() do?  They compile source code; blur images; download files; version data; sort data; search files; visualize information.  In every case a tool() does something that we could do by hand, but without speed or scale.  A tool() reduces time and space, and makes human an inhuman task.

We see some of this in the tool as well.  We call the man who can run 42 Km (26 miles, for my American readers) a hero, while the man who bikes the same distance (or drives it in his convertible) is enjoying an afternoon outside.  In both cases our manual energy is required.  However in the latter case, the bicycle is more efficient at converting it into distance.

I used the word “hero” above, and was intentional about doing so.  In software projects the main goal of tools() is to convert heroic acts into everyday tasks.  Tools(), and those who create them, seek to remove the heroic from the creation of software.  Like all good mythologies, tales of software heroes abound.  We can all name heroes we know, or heroes we collectively know, who have done the impossible with code and computers.

These stories feel good, and connect us to a tradition of storytelling.  They also hide the reality of software projects.  I’ve worked on many software projects, and come to understand that software is never built with heroics–not good software, not large software, and certainly not shipping software.  The mythology of software is that “Mark” built this.  The reality is that a large team did.

Those who haven’t worked on a team before may not have encountered tools() in the way I’m describing.  The team requires tools() in order to depend on what would otherwise require heroes.  While central to the mythology of the battlefield, the hero is anathema to shipping software, and especially software the needs to ship again.

Considering, as I’ve said, that tools() demythologize, that a tool() is a process for removing heroics from software projects, and that tools() are processes meant to shrink size and collapse time, we have to invest in their creation.  Successful software projects are full of tools(), from build systems to test harnesses, profilers to continuous integration systems, linters to compilers.  All of them seek to reduce the need for heroics, to make mundane the exceptional.

To get tools(), one needs to always look within a team for the places where heroics are still required.  The processes that can’t be easily replicated, the knowledge that only a few people have, the process that takes hours or days to complete–these need to get transferred into tools().  We have to invest in this activity, make it part of our culture, resist the hagiography of technical giants and prefer the process-turned-tool that allows for easy repeatability by anyone on the team.

Tools(), like tools, suffer from invisibility when they work.  It is the nature of a good tool that it shouldn’t reveal itself while in use.  As our tools() become better and more sophisticated, we can lose our connection with them, especially their genesis.  If we can’t see our tools() anymore, we also lose sight of their history, the way it used to be, the heroics that were once involved in running 42 Km vs. the bicycle ride (or car trip!) we now enjoy.  The risk is always complacency, the forgetting of the pain and dysfunction that predated the tool()’s existence.

If we don’t make an effort to quantify and present a tool() to our team, if we don’t bear the memory of what came before the tool(), we devalue the culture that brings about tools() in the first place.  When we can say that this tool() is saving us X hours, Y gigabytes, or Z steps, we help elevate history over mythology.  It’s important to celebrate the hero by converting his or her act into a tool() that renders it obsolete.

Tools() need to be valued by the culture of a project.  There needs to be discussion of their worth, and reflection on what they replace.  We need to make time for their creation and maintenance.  We need to continually ask ourselves why we are relying on heroics and heroes.  A tool() should replace a hero, allow Sisyphus to cease pushing the stone to the top of the hill.

June 26, 2012 05:23 PM