Bump Dev Blog

Flock for Android: A Technical Breakdown

Tue, 18 Dec 2012 15:33:16 -0500

This post was written by Indy and Sam, on Bump’s client team.

Flock for Android was just a twinkle in the eye of the Android users at Bump until early October. So when it was time for another hackathon, a few of us got together and threw together an early version of the app you see today. Since then we’ve been iterating and hardening Flock for Android. In this essay we will go through what it took to bring Flock to Android.

An early decision we made was to initially build the app with ICS in mind rather than older versions of Android. We are big fans of UI direction Android has taken since ICS and knew of a lot of great APIs that were introduced with ICS that would make Flock faster and better, and easier to build. Once we had the app working like we wanted, only then did we go back and see how we could make the app work on Gingerbread. This meant that we had to cut out some of the things that make the app super slick, but kept it functional.

We decided not to go back further than Gingerbread. When we looked into it, we noticed market share for OSes pre Gingerbread are shrinking fast. We would open ourselves up for a whole host of bugs if we tried getting the app working with older phones, OSes, and missing APIs. Our small team simply didn’t have time to deal with that.

CoreWhen we started working on Flock for iOS we knew that we’d eventually build an Android version. And we followed the same pattern that we used in the Bump app to share code. We wrote a lot of C. In the Bump app we only had our core networking library in C (we have a custom protocol to talk to our servers), but in Flock we wanted to share code higher up the stack. We wrote all of the “core” logic in C. This is the layer that translates the packets we use to talk to the server into structures that our UI uses.

This C code has been running and been heavily debugged over the months that Flock for iOS has been out, and “all” we had to do was plop that C code into the Android app. Whenever you bring C code onto a new platform there is always a bit of work you need to do. But once that was good to go we had a fully functioning core of the app. All we had to do was consume the structures emitted from it.We use inproc zmq sockets to communicate between this core C layer and the Java UI layer. And the data is encoded using Protocol Buffers. We use exactly this structure for the iOS app, with the UI layer in Objective-C instead.

UIWhile we were in the middle of implementing the iOS app, our design team produced an alternate design of the app, that focused much more on the display and exploration of photos. So when it came time to start implementing the UI, our hackathon team decided to build this design instead of simply porting the iOS UI. This design gives you a feel of exploring a large map of all your events and photos and fit better with the emphasis of larger screen sizes on Android. All the other information is only revealed as needed, giving a much more immersive and engaging experience.

Though we all loved this design there were a lot of challenges involved in implementing it. We only have a time to touch on a few for now.The first challenge was to get the basic layout framework working. Like with anything that has an unbounded list of objects we use the trusty ListView class in Android. Each row in this list is an album. Simple enough, we’ve done this before. However the design called for a horizontal swiping in each row to see the other photos in the album. This seemed like a use of the ViewPager class. Here we ran into our first hiccup.

Most places you see ViewPagers used, like the Play Store app and even the Bump app, ViewPagers page through full screen wide pages. There is one primary view that’s visible and then pages that it dynamically loads on either side of the page depending on which way people swipe. In Flock we wanted each item in our page to be a square with the ability to see a little bit of the page to the right. I thought it was time to start hacking android layouts, or if all else failed rebuild pagers with HorizontalScrollView (sort of how Pulse does it). But after some digging around I ran across a method on PagerAdapter called getPageWidth. This was exactly what I was looking for, though it takes a percentage width instead of a an explicit size, but that’s easy enough do the math for.The second hiccup was how to shoehorn ViewPagers into ListViews. There are two problems with this. View pagers are designed to have vertical scrolling views in them (remember how we mentioned earlier that they are designed to be full screen views), not be contained in views that scroll vertically like ListViews. What this means is that the user must be very exact about scrolling vertically or else the ViewPager would just wiggle slightly left or right instead of letting us scroll vertically.

To solve this problem we basically took the approach described in this stackoverflow answer. We intercepted touches and detected if there was more vertical distance in a scroll than a horizontal. If so then we gave precedence to the ListView over the ViewPager. This simple trick worked out beautifully.Once we had the basic view framework going we ran into our second major challenge. This was also related with trying to shoehorn ViewPagers into ListViews. The core of this problem is that ViewPagers are not really designed to do view recycling, which is essential to smooth vertical scrolling. Dian Hackborn, an Android framework engineer, had an exasperated reply when someone asked for help in doing this online:

How exactly do you recycle the list view items correctly? This is… crazy. - Diane Hackborn

Because of this we initially thought we were heading down the wrong path, but there were just too many things we got for free by using ViewPagers as ListView items that we decided to be a little crazy.

To get around the lack of view recycling we subclassed PagerAdapter and implemented recycling ourselves. The exposed methods are actually almost identical to a standard BaseAdapter (getView, getItemViewType etc.). We used a map of view type to queues of available pages to store recycled views. Whenever destroyItem is called for a particular view type, we add the corresponding view to the queue for that view type. Whenever instantiate item is called, before actually inflating a new view, we first check to see if we have a saved view in the corresponding queue in the map. While we got reasonable performance using this schema, it wasn’t sufficient. The order of the pages and the types of the pages in each row in Flock is determined by the number of photos in an event. Since most events have different numbers of photos, each row had a different sequence of view types. As the ViewPagers and PagerAdapters were reused during the user’s first few vertical swipes, the adapters were continually being asked to load new view types that other adapters had inflated, but the current one had not. Until every adapter had been asked to inflate a few of every view type we were inflating a lot of views, so there was a prolonged period of poor scrolling when users first opened the app. Since every adapter kept two or three views of each view type around, we were also using a fair amount of memory just keeping views around. The final step to help alleviate over inflating views and reduce memory usage was keep a single global map of view type to queue of views, rather than having each PagerAdapter have their own. Sharing views for pages across rows meant that we didn’t need to inflate as many views for each type and similarly didn’t need to keep as many views for each type around in memory.

Along with some timely help from Romain Guy’s blog on android performance and our own efforts at view recycling, we’ve been able to get good scrolling performance despite having complex row views displaying large images with very little padding. There is always room for improvement and we have a few ideas in mind to eliminate the last few scrolling hiccups for users with lots of events in a future version, but on the whole we’re pretty happy with the performance.Background Location
One of the really interesting things Flock does is find out the friends you’re hanging out with to recommend exactly whom to share photos with. To achieve this we need to know where a user is without having them open the app. This is where the differences between iOS and Android API philosophies are really obvious.iOS gives apps a couple ways to do this as very high level APIs. One is to register for SLCs (Significant Location Updates) and a second is to register geo fences (region monitoring). Once you register for these you are at the system’s mercy to wake up your app when it thinks it’s right for you. Currently Flock for iOS relies primarily on SLCs, however SLCs didn’t behave the way we expected them to since they were designed for the opposite use case: moving a “significant” distance but not necessarily settling into a new place, which is what Flock needs. SLCs also come at a significant user perception cost because if your app registers for them, iOS will always show a solid location services arrow. Always (Even though iOS’s internal Traffic monitoring tools on your device use the same APIs but don’t show the arrow). This gives users the perception that the app always has the GPS on and is constantly consuming large amount of battery power, which is not really the case.
Android on the other hand generally gives you access to much lower level data. You can register to listen to location updates on different location providers (GPS, Wi-Fi, Cell Tower Triangulation), and you can set up your app to be woken up at a certain time in the future and check these services yourself. This low level control gives developers a really easy way to shoot themselves in the foot by mistakenly asking for location too often (thus causing massive battery drain). But it also lets us be much more in control with our algorithm, giving us simpler ways to get to the information that we really want.All in all, it was a blast getting the Android version of Flock ready for prime time. We are super excited to have the app out there for general public to use, and drastically increase the number of people you can share photos with on Flock.

My Summer Internship at Bump

Thu, 27 Sep 2012 14:29:00 -0400

This post was written by Andrew, a summer intern on Bump’s client team.

Deciding where to intern is a big decision, or at least it was for me. Having never interned at a company before, I didn’t really know what to expect but knew what I wanted. I wanted to work on interesting problems, have freedom to create, and work with great people. With these considerations in mind, I chose to intern at Bump.

Work
I came into Bump as an Android developer. Prior to Bump, I had some experience with the framework – I’d published an app. However, I still had much to learn (better realized after my internship).

One of the most integral skills to software development is being able to reason at multiple levels of abstraction, and this is a skill my experience at Bump helped me hone. The Bump Android app is complex, but necessarily so. The app makes use of many of Android’s facilities, including that for embedded native code. The app must take into account the interaction between UI actions and network events, and does so nicely. When working on the Bump app, the professionalism with which it was created is noticeable. Looking back, I think my appreciation for the app’s structure is a testament to what I learned over the summer regarding modularity and reusable code design. Sure, anybody learns in school how to write generic code inside a few source files, but what about writing a component composed of many source files which must interact with a much larger machine?

Another key skill I developed at Bump was working on a large codebase with other people. I am now infinitely more familiar with git than I was at the start of the summer, including managing projects with submodules. I think the day-to-day aspects of software development such as version control, setting up build environments, and team communication are under-emphasized in school and am glad for the experience I gained at Bump.

Freedom
An aspect of the work I did at Bump that I really enjoyed was the freedom I was given. The nature of the problems I solved at Bump was not “do this in O(whatever) runtime”. No, I was tasked with solving real problems the company faced – high level problems – and was given guidance in solving them. I found this freedom to be pervasive throughout my internship experience. For instance, I needed a lightweight way to log data from tests I was running and, even though I’m an Android developer, was able to play around with CherryPy to write a logging system. If I had questions, I was able to ask one of CherryPy’s lead developers – he works at Bump.

Freedom to do things the way you want is embedded into the culture of Bump. The Android developers use IntelliJ because they like its memory usage and code completion better than Eclipse. However, if you want to use Eclipse, you can. Want to run Eclipse inside an Ubuntu VM? Go for it. How about forgoing the whole thing and just using vim and a terminal? Your choice.

The openness and willingness to try new things as a company is one of my favorite aspects of Bump.

Life
As I’m still in college (Stanford ‘14), life at work was a major factor in deciding where to intern – Bump does not disappoint. I think it’s pretty cool when you can play Berlin-style ping pong and sing karaoke at work. The people at Bump are great and fun to be around. While people take their work seriously, they don’t take themselves too seriously.

Speaking of ‘seriously’, you may be wondering what the hours are like. Most people don’t show up until around 10 which was great for me. People generally leave around 7 but there is no set time. Personally, I’d usually leave around 8, but that was voluntary – I just figured that experiences like these don’t come too often and wanted to make the most of it. However, the culture is flexible – a few days I had to leave around 5:30 and would, no questions asked. The emphasis at Bump is on the actual work you do, not the time. That said, if you don’t want to work, don’t work at Bump. At Bump, you will work but you’ll have fun doing it.

Food
Food was great – would eat again.

At Bump, you’re surrounded by food. The company has a snack bar and refrigerator of drinks which get replenished weekly. You can have as many snacks/drinks as you want whenever you want. The company provides two catered lunches per week. Lunches range from Indian food to Pakistani to Jamaican. There were times when I would have preferred a burger or pizza, but I took the opportunity as a great time to expand my palate. Most importantly, when Bump gets food catered, there is a lot of it. There are usually leftovers which is awesome for the days following catered lunch days. Bump’s also really close to Castro Street which has more food than you can eat in a summer (I tried) and there is almost always a group willing to go grab something. For the company I’m not sure how the unlimited food model works – my productivity definitely went down during the post-lunch food coma, but for me it was great.

Should I Work There?
If you’re reading this, you’re probably asking yourself that question (otherwise, you have a lot of free time and/or enjoy reading intern blogs).While we both know there’s no definitive answer, Bump has my recommendation. I had a great time interning at Bump and the rest of the interns my year seemed to have very positive experiences as well. Ultimately, you’re going to have to go with your gut. I did, and it was one of the best decisions I’ve made.

Martin Gardner had a timing attack for your mind (1956).

Fri, 17 Aug 2012 10:12:00 -0400

This post was written by Seth, Bump’s resident iOS Magician.

Martin Gardner was a math and science writer, who also was a magician. In one of his books, he teaches a magic trick that bears some similarities to a more recent technique for stealing passwords from systems that are using an insecure HMAC checking algorithm.

So, let’s bring together HMAC security, mathematician Martin Gardner, magic, and a timing attack on your brain (and throw in app virality at the end). Do I have your attention yet, Hacker News?

What is a Timing Attack?

A timing attack uses information leaked by a computer system based on the amount of time it takes for something to happen. For example, let’s imagine I want to compare two strings for equality (say to check a password (which you would never want to do!)): “Hello World!” and “¡Hola Mundo!”. A simple algorithm could compare letter by letter, and return false as soon as one character doesn’t match. In this case, no characters match, so it would return pretty fast. If I were comparing “Hello World!” to “Hello Wormy!”, the algorithm would have to compare 10 characters, and so would take 10 times as long (factoring out the parts of the function that are run on every call). Using this timing information (on a sufficiently slow machine), you could uncover the password, one letter a time by simply iterating over each possible letter until the algorithm took a little longer, and then proceed to do the next character in the password.

This overly-simplified timing attack is similar to a real potential attack to using HMACs.

Magic Trick?

Martin Gardner described a timing attack on your mind in his book ‘Mathematics, Magic and Mystery’ in 1956. It is a magic trick called “Divining a Number” which involves discovering the number that your spectator is merely thinking of.

You may have heard of simpler magic tricks where you ask someone to think of a number, perform some math, and then can tell them the number they end up on. This magic trick is different. You tell them the number they originally thought of, and they never tell you any of the results of their mental math.

You ask them to think of a number, have them do some mental math, without saying anything, and then you know which number they thought of.

You discover the thought-of-number by using a timing-attack like process on their mental math. A simple example:

Think a number: 8 or 9 (a simple choice of two numbers).

Now, divide that number by 2.

To be fair, you are probably better at mental math than the average person by virtue of the fact that you are reading this article, but it is signifcantly easier for most people to divide an even number by 2 than an odd number. So, by watching your face as you did this mental math, I could tell you which number you were thinking of. If you were really fast, you were thinking of 8, slow means 9.

If you add a few more mental math timings, you can expand the original range of choices from 2, up to 10. Martin Gardner describes the whole (rather complicated) procedure in his book.

Martin Gardner didn’t invent the technique (he heard about it from Edmund Balducci), and it is based on several tricks found in “The Magician’s Own Book” published in 1862! Timing attacks go way back!

App Virality?

I’m a magician, and also a software engineer at Bump. Part of my job is to explore using the theory behind magic tricks (including mental timing attacks) to improve our app’s virality. If you want to learn how, or if you want instructions for the full magic trick, e-mail seth@bu.mp !

A Bump Up - Reflections on Startup Life

Sat, 16 Jun 2012 14:22:00 -0400

This post was written by Greg, a summer intern on Bump’s server team.

After a solid first internship with Amazon’s Kindle division in Seattle last summer, I wanted to explore the opposite end of the tech company spectrum for my second internship, so I could make more well-informed career decisions after graduation. Bump caught my attention very early in the internship search. It was a small company working with open source software developing a truly one-of-a-kind product that makes life easier for tens of millions of users. A userbase of that size provides a great opportunity for machine learning, exposure to real-world architectural and scaling challenges, and a guarantee that my work would have a big impact.

My interest in Bump only grew during the interview process. Bump’s programming challenge and interview questions were practical and engaging, which was a refreshing change from the slew of linked-list manipulation, string-integer conversion, and fizzbuzz questions I had been answering in other interviews at the time. Before I knew it, I was on my way to California for the first time, and to the heart of Silicon Valley at that!

The first week was a bit overwhelming; it took a while to get used to the unstructured start-up environment and to learn who to ask for help. Once past these initial roadbumps however, I’ve gotten the chance to explore a variety of cool projects with the server and data teams and am still learning a ton everyday. In reflecting on my time here, I’m very grateful for a few qualities of this internship that exceed my experiences from last summer, especially these three things:

Important work. While the project I worked on at Amazon was a neat idea that would have improved the Kindle experience for tens of thousands of people, it was more of a nice-to-have feature than anything else. I found out later that, unfortunately, it was shelved after I left because there were more valuable things to work on. This is not an issue with Bump. Because the company is so small and the userbase is so large, everyone’s work is critical (even interns’) and there is no shortage of important work to take on. The effort of a single person here can have a very visible impact on the product, and consequently, on the lives of millions of people. It’s exciting that my work on debugging tools and analytics will directly improve the service and experience for all of Bump’s users.

Speed. I feel like I’ve accomplished more at Bump already than I did all of last summer. The pace throughout the company is very fast, and it works because everyone has a “get things done” mindset. There aren’t any obligations to distant departments or chains of managers to go through, and everyone in the company is within a 10 second walk of my desk, making it easy to ask questions and clear obstacles. The speed is a great motivator, as well, because progress is evident and exciting as we rapidly approach our goals.

Fun culture. Amazon had very cool weekly tech talks by senior engineers and also brought in various performers and artists (I even got to meet my favorite author, GRRM!). Aside from these events though, the majority of my work culture experience at Amazon was dominated by the company policy of frugality. I have a lot of respect for that policy and all the work Amazon puts into lowering prices and improving service for consumers, but I feel that having a much brighter, more open office with a few games and snacks has greatly improved my mood while at the office. I appreciate that — when I get stuck on a problem — I can get up, be active, and talk through the problem with a coworker over a game of ping pong. By the time I return to my desk, I can see the solution to the problem or I will at least have a new perspective or approach for tackling it. Little things like this, plus organized company events such as trivia night and ultimate frisbee matches against other startups have made the entire internship experience more enjoyable.

I’ve found that qualities such as these are very important to me not only today as an intern, but also as an eventual career-seeker. Bump has been successful at not only attracting smart, like-minded people, but also cultivating a positive and playful work culture where everyone motivates each other to build the best product they can imagine. Needless to say, I’m excited to be a part of this team and to continue building and learning here for the rest of the summer!

A Bump Up

Sat, 16 Jun 2012 13:57:00 -0400

This post was written by Greg, a summer intern on Bump’s server team.

Internships are such a fantastic opportunity to explore different companies, work cultures, and cities that I sometimes wish I could forever do internships, trying and learning a little bit of everything, everywhere. Alas, there are only a couple summers as an undergraduate to be an intern, so I wanted to pack in as much learning and fun as possible.

After a solid first internship with Amazon’s Kindle division in Seattle last summer, I wanted to explore the opposite end of the tech company spectrum for my second internship, so I could make more well-informed career decisions after graduation. Bump caught my attention very early in the internship search. It was a small company working with open source software developing a truly one-of-a-kind product that makes life easier for tens of millions of users. A userbase of that size provides a great opportunity for machine learning, exposure to real-world architectural and scaling challenges, and a guarantee that my work would have a big impact.My interest in Bump only grew during the interview process. Bump’s programming challenge and interview questions were practical and engaging, which was a refreshing change from the slew of linked-list manipulation, string-integer conversion, and fizzbuzz questions I had been answering in other interviews at the time. Before I knew it, I was on my way to California for the first time, and to the heart of Silicon Valley at that!

Important work. While the project I worked on at Amazon was a neat idea that would have improved the Kindle experience for tens of thousands of people, it was more of a nice-to-have feature than anything else. I found out later that, unfortunately, it was shelved after I left because there were more valuable things to work on. This is not an issue with Bump. Because the company is so small and the userbase is so large, everyone’s work is critical (even interns’) and there is no shortage of important work to take on. The effort of a single person here can have a very visible impact on the product, and consequently, on the lives of millions of people. It’s exciting that my work on debugging tools and analytics will directly improve the service and experience for all of Bump’s users.
Speed. I feel like I’ve accomplished more at Bump already than I did all of last summer. The pace throughout the company is very fast, and it works because everyone has a “get things done” mindset. There aren’t any obligations to distant departments or chains of managers to go through, and everyone in the company is within a 10 second walk of my desk, making it easy to ask questions and clear obstacles. The speed is a great motivator, as well, because progress is evident and exciting as we rapidly approach our goals.
Fun culture. Amazon had very cool weekly tech talks by senior engineers and also brought in various performers and artists (I even got to meet my favorite author, GRRM!). Aside from these events though, the majority of my work culture experience at Amazon was dominated by the company policy of frugality. I have a lot of respect for that policy and all the work Amazon puts into lowering prices and improving service for consumers, but I feel that having a much brighter, more open office with a few games and snacks has greatly improved my mood while at the office. I appreciate that — when I get stuck on a problem — I can get up, be active, and talk through the problem with a coworker over a game of ping pong. By the time I return to my desk, I can see the solution to the problem or I will at least have a new perspective or approach for tackling it. Little things like this, plus organized company events such as trivia night and ultimate frisbee matches against other startups have made the entire internship experience more enjoyable.

From MongoDB to Riak

Mon, 14 May 2012 17:09:00 -0400

This post is written by Tim on Bump’s server team

At Bump Technologies, we recently completed a significant database migration from MongoDB to Riak. Almost all of our users’ data — the lists of people they’ve bumped, communications sent and received, handset information, social network OAuth tokens, etc. — had been stored in MongoDB, but if you open the app today all of these interactions will be backed by Riak.

A few months ago we had migrated our datastore containing individual communication contents from MongoDB to Riak, so these databases and general migration techniques were not entirely foreign to us, but any time you have to move data while keeping the system online, it’s a unique and complicated challenge.

Development obstacles

The largest development hurdle was dealing with the structured key-document nature of MongoDB vs. the key-value nature of Riak combined with the latter’s eventual consistency. At Bump we use Google’s Protocol Buffers extensively, for serialization to Riak and message passing between the Bump client and the server, and between server application nodes. In addition to being an efficient encoding, it also enforces types and provides for backwards- and forwards-compatibility. MongoDB’s key-document knowledge and consistency model allow it to support, e.g., atomic list appends, which we would use to add to a user’s list of communications when they sent one.

Riak does not have any internal knowledge of how the Protocol Buffers we are storing are formed, and cannot guarantee the same kinds of consistency. Instead, we have to grab the value for the key, update it ourselves, and put it back in. That’s easy (if a bit more verbose than what MongoDB does), but the difficulty comes when dealing with concurrency. One of our app nodes could get a user’s record, then append the communication the user just sent, and save it to Riak. The database, being eventually consistent, cannot guarantee that when another app node gets the record it contains the communication just written. If the second app node then makes changes and overwrites the key with their new value, data will be lost.

Riak is pretty smart, though, and will save both of these values. Upon the next get, it will return the two valid values (if, that is, the node has received both!), leaving it up to the app to deal with the multiple truths. With appropriate logic (set unions, timestamps, etc) it is easy to resolve these conflicts, but dealing with this everywhere a database fetch happens is cumbersome. Instead, we’ve written a thousand-line Haskell program that acts as an interface between the app nodes and the database, which exclusively deals with conflict resolution and sibling merges. Every Riak interaction is done through this tool that we’ve dubbed /magicd/, which guarantees the app nodes always see a consistent truth. This obviates the largest pain point of eventually consistent databases.

After evaluating our migration options, we decided to approach it on-the-fly. The other significantly considered technique was migrating in parallel; everywhere Mongo was written we would append a write to Riak, and everywhere a Mongo read was done it would be prefixed with a read from Riak, falling back to Mongo if the data had not been migrated. This two-database parallel approach was appealing in that if something horrible happened during the transition, we could revert to pre-migration behavior (i.e., drop all of the Riak code), and the data would be consistent and up-to-date in Mongo as before.

The difficulty with this approach was that, despite almost all of our database interactions being encapsulated in a class, there were a few scattered circumstances in which the database was written to outside of the class. Those instances, of course, were only a grep away, but the significant amount of code / logic duplication for those special-cased interactions was a deterrent.

Instead, we decided to migrate all of the users’ data on the fly on login. With the first packet received from the handset, we block all other communication until the user has had all of its data written to Riak. It then proceeds as normal, with all other database interactions in the server codebase having been converted to use Riak exclusively. The upside of this compared to the previous parallel approach was simplicity; s/mongo/riak/g and call it a day, instead of handling two data code paths. The downside was that a rollback to Mongo in case of significant errors could lose data, either temporarily if the new data written to Riak was recoverable, or permanently if not. Despite the downside, with extensive testing we were able to ferret out any bugs that would result in data loss, and suffered none on roll out.

Post-deployment obstacles

Significant testing before rollout could nevertheless not definitively show us all of the bugs that existed. Specifically, we encountered a few errors that were only made evident with production data and traffic.

The first issue we saw after deployment was one of our Redis instances complaining about hitting its maximum connection limit. We use Redis throughout Bump’s backend, performing a number of duties including queueing, session caching, and even geolocation mapping. Using it as a networked locking mechanism for migration purposes was a natural progression. The code we wrote, now included as part of the diesel async framework we use, did not appropriately close the connection in some circumstances. With that fixed, the Redis connections dropped to a normal and expected number.

The second class of issues we encountered was a product of the relaxed schema of Mongo and the strict serialization of Protocol Buffers. There’s no way to enforce types for values within a Mongo document; an integer may be stored as an integer, or a string representation of the same, for example. Depending on the leniency of application code, this fast-and-loose typing may be acceptable.

Protocol Buffers are much more strict about the types they accept. If one tries to assign a string to a value defined as an int32 in the protobuf definition file, it will refuse to be serialized. This manifested itself in our code when we tried to serialize the Apple Push Notification Service token; in some circumstances, it contained characters that broke when being parsed as a Protocol Buffers string; changing that type to bytes fixed the issue. This modification of the definition file preserved backwards-compatability because strings and bytes are stored in serialized format in the same way.

Unlike JSON/BSON where there is no defined schema and one can throw in any values they desire, Protocol Buffers can specify required fields. A object that must have a user_id, for example, cannot be serialized without that field set. This does wonders when doing client-server communication or serialization to a database, for contracts can be made and consistently followed. During the migration, there were a number of fields that should have been set in Mongo but were not. Luckily, they were comparatively menial values — last seen Bump version number and locale for a user, for example — but their lack of existence in Mongo caused their migration pipeline to break. Setting empty locales and 0.0.0 version numbers was deemed a satisfactory resolution considering the importance of the data.

With the above issues solved, our Python processes stopped spewing hundreds of exceptions, and things on the app node side seemed to be cooling down. At this point, we turned our attention to Riak load; we started to see latencies rise to concerning levels. In addition, we started receiving user reports that database-heavy operations were acting slower than they had prior. Further investigation showed that our magicd processes were pegging the CPU, so as a temporary fix, we increased the number of threads the process would spawn. Benchmarking revealed that magicd was serializing and deserializing every protobuf that went through it; this was only necessary when siblings existed and needed to be resolved, so fixing it so deserialization only occurred when required reduced load to a more manageable level.

While this helped magicd CPU requirements, we were still seeing elevated Riak latencies. Investigating the logs showed some particularly abnormal behavior on one node in particular; upon sshing in and poking around, it became apparent that one of its disks had failed. Shutting down the riak daemon on that node caused to latencies fall back to normal levels. After getting the disk issues resolved, we were able to bring the node back online. Thankfully due to Riak’s destributed nature, a missing node in the cluster was not a problem; we still saw great performance.

Post-Mortem

We decided to move to Riak because it offers better operational qualities than MongoDB. Despite the migration hiccups, we’re pleased with the process and the result. Now we no longer care if one of the nodes kernel panics in the middle of the night; as has happened a few times already. Nagios will email us instead of page us, and over coffee the next morning we’ll fire up IPMI, reboot the machine, and Riak will read-repair as necessary. No longer will we have to do any master-slave song and dance, nor will we fret about performance, capacity, or scalability; if we need more, we’ll just add nodes to the cluster.

If you have any questions about our experience, or want to work with us on this joyful stack (we’re hiring!) — shoot me an email: timdoug@bu.mp.

git for dropbox users: Don't be afraid!

Wed, 29 Feb 2012 16:44:00 -0500

git is awesome

We use git at Bump for all sorts of things. Recently, we’ve started converting our designers and product managers. You can think about git like Dropbox: it takes files on your computer, puts them in the cloud, and lets other people work on them too. But it has some pretty rocking advantages:

Collaboration

Like Dropbox, git lets multiple people work on the same files at the same time. This is great when many people are working on a document together, or a group of people are creating assets for a website. Everyone can be contributing to the same folder and files, without fear of overwriting eachother’s work, something Dropbox can scew up.

Time Machine

Even if you don’t have a team of people, git has some awesome-sauce. It can act like a time machine, allowing you go and see what you did last week, or even last year, help you recover files you deleted, and keep track of progress over time. It has powerful tools to let you see what you’ve changed over time.

And More

git is also great for backing things up, so you don’t lose them. It can let you learn about how you work: when do you do your best work? How often do you change everything? There are tools to show you visual differences between versions of images, and all sorts of things.

but

git is awesome… but conceptually it’s a leap from most people’s mental models of what a file is.

There are so many wonderful reasons for non-programmers to use git when working with other people (especially programmers) and I’d like to convince you that there are even more awesome rewards down the way of git, but I won’t do that today.

My goal today:

Provide an analogy to Dropbox to help designers and product managers understand the mental model of how git works.

It might help you to think about git like Dropbox to start. It’s like Dropbox but with a whole lot more steps. Have faith (for the moment) that the more steps are worth it, and I’ll talk you through the differences.

First, a quick overview of how Dropbox works: many people can be working on the same file on their own computers, and magically they all get each other’s changes. Behind the scenes Dropbox is doing all sorts of things to make this work. With git, you end up helping do some of the behind the scenes work, and git thanks you.

As easy as 4… 5… 6!

Here is a more explicit model of how Dropbox works.

You install Dropbox.
Someone shares a folder with you.
Your computer downloads the contents of that folder from the “Cloud”
You make some changes to a file in the folder and save it.
Your changes get sent up to the Dropbox “Cloud” immediately
Everyone else gets your changes downloaded to their computers immediately.

Yay. This is just like git! Only git lacks some of the “immediately” magic. Steps 4, 5, and 6 don’t happen automatically: you have to tell git when you want it to do each step. Let me break it down for you.

4) Saving… are you sure?

The first difference is that git doesn’t assume that you want to send every saved change to the cloud. It makes you be explicit about what you want to send to the cloud.

Whereas Dropbox automatically uploads everything when you save it, git adds a new step: commitment. After you save a file, you must tell git that you are “committed” to this change.

To do this, you do what is called “staging.” This is really just telling git which changes you were sure about. This way, git doesn’t waste other people’s time if you make a draft you don’t like.

5) Uploading

Second difference: Dropbox uploads your files immediately. (It tries to anyway). git isn’t quite as eager to get things into the cloud.

In git, uploading to the cloud is called “pushing.” You are literally pushing the changes from your computer up in to the cloud. Only then will other people be able to download them.

6) Downloading

The next difference is downloading. Can you spot the difference? Why, that’s right! While Dropbox downloads the new changes automatically, git waits for you to tell it too.

You must ask git explicitly to “pull” the changes down from the cloud to your computer. This gives you the latest greatest copy of the documents, including everyone else’s changes

Stay strong! Keep faith that even though there is seemingly less magic, and more work, there are reasons for all this mortal toil [sic]!

Ordering, or 6… 4… 6… 5…???

git requires you to operate with a slightly different order of operations because it is more manual (slightly less magic).

Before you start changing things, you want to make sure you have the latest and greatest version with everyone else’s changes. That means the first thing you want to do is download the latest stuff from the cloud (i.e. do a pull).

Now that you have the latest, you can change it and edit it to your heart’s content. When you are happy with your changes you are ready to tell git you’re happy (i.e. commit).

You can also rinse lather and repeat step 4 many times, committing small changes before going to the next step.

But wait a second! What if Brian in marketing changed the document WHILE you were working on it? Because git doesn’t automatically download the changes, your computer (and you, by extension) doesn’t know about the new changes. So this adds another step, redownloading (i.e. doing another pull!)

Okay, now git knows about your changes, it knows about Brian’s changes, and the only thing left to do is… Push! Finally we can upload our changes to the cloud for everyone else to see and look at.

That’s the gist of git.

It’s like Dropbox, but you have to tell it to download, save, download, and upload each time. (Pull, stage & commit, pull, and push)

If you are digging it so far, let’s tackle some Advanced differences.

Advanced Differences

There are some other model differences between git and dropbox. The first are conflicts: what happens when two people change the same file. Another big one is the fact that git is “distributed,” i.e. there is no company holding the cloud for you… there can even be multiple clouds.

Here are some quick overviews of salient differences.

Time Travel

Every commit makes a little tick on a timeline. git keeps track of EVERY little tick. That means you can zoom back in time, and watch as your document changes from day 0 to present. You can see every change, along with who made it, and what there “commit message” was. Pretty cool.

git has multi-player undo You can find out exactly who made the typo, and when and then fix it really easily. (git has really awesome tools for this too).

Conflict

What happens when Brian changed that file in the cloud, while you were working on it on your computer? Dropbox takes care of it magically in the cloud, now… it’s your job. The second step 6 (re-pulling changes from the cloud) in your work flow is when you are most likely to have “merge conflicts”.

Usually git is pretty magical and figures out what to do if both you and Brian were editing the same file. For example if he changed the title of the document, and you added some foot notes, git will figure it out. However, if you change your salary to $350,000, and Brian changes it to $40, git isn’t going to know what to do.

If Brian pushed his change first, git will make you fix it. It will say: “Brian said you make $40 an hour, but you said you make $350,000, who’s right”. Your call. Fix it, save it (commit it) and push it it! (but do pull it in case Brian caught his error and changed it to $400,000 while you were typing)

Meta Comments

Every time you save (commit really) some changes, git asks you for a “commit message”. This is couple of sentences that describe the changes you made: a brief overview so that someone could review the list of changes to this document, without having to read the whole thing.

Style

Because git is a little more manual, there are a few stylistic changes suggested to make everyone happy. First of all, small changes. You should commit every time you finish changing a coherent piece. I.e. if you are changing your salary, and Brian’s, first make one commit with your increase and a commit message like “merit based salary adjustments”, then adjust Brian’s with a commit message of “balance the budget”. Good (informative) commit messages are very appreciated in git land.

Due to the nature of fixing conflicts, git also prefers text. Images are much harder for it (and you) to merge if both people change at once. So whenever possible use a text format (like html or markdown).

Fin

Well, I don’t know if I convinced you that you HAVE to use git, but hopefully you understand it a little better now. If you are already using it, hopefully now it makes a little more sense. At the very least, you are a step closer to being a friend and co-conspirator of git.

Easy partial screen modals on iOS

Tue, 18 Oct 2011 15:12:00 -0400

This post was written by Anders from Bump’s iOS team.

Say you would want to present a modal view on an iPhone but you don’t want it to be full screen. What do you do? Well, there is always the option of just not using presentModalViewController:animated: and performing the animations yourself which also in turn means maintaining view and controller hierarchy yourself.

You could of course subclass UINavigationController and encapsulate all that complexity in a subclass. That would of course mean that you would need to update your code in all places that use the UINavigationController. Another similar approach would be to subclass UIView and override setFrame: to not allow the frame to be set to full screen. Again, that would mean updating all relevant views to subclass your new UIView subclass with the setFrame: logic.

A different approach which lets you maintain your code pretty much unchanged requires you to change the way UINavigationController itself behaves. The following solution lets you set the frame of your UINavigationControllers view which will then be maintained even if a modal view controller is presented.

All you need to to is call a static initializer method on this new NPNavigationController class:

[BPNavigationController setup];

Now you can set the UINavigationControllers view frame. Like so:

navigation.view.frame = CGRectMake(0.0f, 20.0f, 320.0f, 440.0f);

If at any later point you present a modal view controller in a navigation controller that modal view controller will only be as big as the frame of the navigations controllers view.

How is this magic achieved? Behold:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

// BPNavigationController.h

// StatusBarSize

// Created by Anders Borch on 10/16/11.

#import <UIKit/UIKit.h>

@interface BPNavigationController : UINavigationController

/*!

@method setup

@abstract Setup UINavigationController to handle non-fullscreen modals.

@discussion

This will replace the implementation of presentModalViewController:animated: with a new implementation

which retains the original frame of the UINavigationController#view and pushes a

+ (void)setup;

@end

Who said Android apps can’t look good?

Fri, 30 Sep 2011 13:48:41 -0400

This post was written by Indy from Bump’s Android team.

Today we are releasing Bump 2.4 or what we have been calling the Visual Refresh. Earlier this year we hired our first (and so far only, we’re hiring) badass visual designer, Shona Dutta. Since then she’s been on a crusade to make our apps and everything else we present (including the recent infographic) look as awesome as our technology.

So when it was time for the Android app to get it’s much needed facelift we wanted to jump on it right away. Unfortunately we were still finishing up some other major features in the app, so our team’s summer intern Will Whitney, cut a branch and started churning away (If you’re an awesome Android dev, this is why we need you, there are too many things to get done and we are currently only two people). He left about a month ago, and finally in the last two weeks we had the time to finish up the work he started.

In the last 10 months we’ve learned a lot about the ins and outs and how to build a great app on Android. One of the things we learned quickly was people rightfully bagged on Android apps not looking as good as their iOS counterparts. But the reasons they pointed to were just plain wrong.

It’s a common misunderstanding that Android apps are harder to program than iOS. People say it’s because the lifecycle behind Activities (Android’s rough equivalents of iOS UIViewControllers) are hard to work with. Not true, I think reading through the documentation on that once should clear up a lot of confusion. People say it’s because the UI code is a bunch of messy xml. Not true, yes it’s xml, but it’s xml used in the way it should: declaratively and in a DRY manner. The xml based layouts are what make UI programming quite a lot nicer: one because it forces view code to be purely view code; two because simple parameter tweaking can produce great results.

In fact I joke with our iOS team that android UI is ridiculously fast to iterate on. Case and point is the global network status bar we have on top of the new app. Whenever the network goes down or is unavailable on any screen of the app a thin black bar rolls down above the whole UI (squeezing the content area below it) letting the user know what went wrong. This simple squeezing of the UI and adding a view on top of every screen took less than two hours to put together.

The real reason Android Apps historically haven’t looked good is because people haven’t put in the same care in building those apps. From a business sense that made sense in the past. Android has never been the primary platform for app users. But the rate of growth of the platform is phenomenal, and even if you believe that in general an Android user is less likely to use apps than an iOS user, the raw numbers should shock you into putting the same care into your Android app as your iOS app.

So it’s finally here: Bump 2.4 for Android. An App that finally looks as good as it works.

See the full gallery on Posterous

Bump's Visual Refresh for iOS

Mon, 27 Jun 2011 15:24:00 -0400

This post was written by Shona from Bump’s design team.

Bump is simple. Right? You bump, and data is transferred from one phone to another. It can be anything from your contact info to photos to apps, but the premise is basic and totally functional. This was what I figured back when I started working on the visual redesign of the app.

Three months later, my opinion has changed (kind of a lot). Bump is an incredibly complex, nuanced app that calls for a whole lot of visual interplay between screens, between elements on those screens, and across the whole application. It’s got hundreds of permutations and possibilities in what can be on the screen at any time. Fortunately I had the help of Abby, our UX designer here at Bump, when I undertook the work of giving Bump a visual “refresh.”

Our choice was to make Bump more tactile. The app is based on the physical bumping of two phones, and there’s something really magical about highlighting a physical interaction in a world where you can talk with someone without ever seeing them, and show them photos without standing next to them. We love the fact that people get a kick out of making real contact when bumping. To carry that further, I decided to explore textures, shadows, and layering in the new design. These are things we see in real life and that make the app look touchable.

Our homescreen background is paper-like; the buttons & graphics are tending toward a letter-pressed style; both of these are enormously tactile in appearance. Subtle dropshadows and glows give buttons, notes and bars the impression of popping up and out of the screen. This is all stuff we see everyday (just look at the pages of a book or the buttons on a TV). These details come from things we interact with and that we know how to use.

At Bump we’re trying to capitalize on the intuitive understanding that people have of objects, buttons, and icons, and use them in the app to help our users get where they want to go. In the end it’s all about making Bump easy, and a pleasure, to use.

As with any large undertaking this project takes time. We’re only part-way finished with the redesign and we’re already looking ahead to a larger overhaul. But in the meantime I’m enjoying the new look we’ve injected into the app. It’s a huge step forward in making Bump even more a pleasure to use.

Want to work with Shona and the rest of the Bump team? We are hiring: http://bu.mp/jobs.

How we use Scala in Bump for Android

Mon, 13 Jun 2011 17:06:00 -0400

This post was written by Will from Bump’s server team.

Since Indy’s post and Michael and Indy’s talk at Scala Days, many of you have been asking about how we used Scala for the Android version of Bump. I’ll try to cover a little of the why (without getting to deep into the Scala v. Java battle) and then get into the how. But first, a little background.

As with many startups, we’re primarily limited by the number of developers we have working on projects and our Android team was no exception. After releasing 2.0 on iOS, Jamie and I (the server team at the time) had plenty of things we wanted to do to the server, but none of them seemed as pressing as bringing Android up to parity with iOS. Knowing the network stack better than most, we decided that Jamie and I would write the Service while Jake K. and Indy would switch over from iOS and write the Application layer. The final decision ended up being that the Service would be in Scala, whereas the Application would be in Java. So then, why did Jamie and I choose Scala?

Why Scala?

It has a lot of functional constructs we’re used to and like. We use Haskell at Bump for various projects and we’re both heavy users of the list comprehensions, first-class functions and other functional constructs that are available in Python. Scala let us express problems in a way we were used to.
Scala works very well with existing Java frameworks. Android is, of course, one big Java framework and we were going to have to interface with it a lot. A number of people asked, given the above, why not use Clojure—this is why.
An Android Service can withstand higher latency. Scala runs a little slower than Java (I’ve heard about 6x in benchmarks, but I’d bet it’s worse in Dalvik), but this was something we could tolerate. If an application takes 50ms extra to render a cell in a list, the user will most certainly notice. If, however, the Service takes 50ms more process something it had to fetch from the network, nobody cares—it took 10x longer to do the network fetch anyway.

Why not Scala?

With all that out of the way, it’s worth taking a minute to talk about some of the problems we ran into.

Scala does have public member variables. To support some of the syntactic changes that Scala makes, it replaces public member variables with accessor functions. In Scala, when you do something like foo.bar, it implicitly calls the function bar() (that returns the value of the private member variable bar) on foo. Java, of course, doesn’t do this. So, when you are using Scala classes in Java code, everything becomes a function and you have to explicitly refer to foo.bar() everywhere. An easy problem to deal with, but it requires the programmer to remember whether something they are using is a Scala class or a Java one.
The above creates a more subtle problem with Android that I wanted to call out specifically—Parcelables. Android’s Parcelable interface depends on being able to access the public member variable CREATOR on a class that implements Parcelable. If you create the class in Scala, then, of course, CREATOR ends up being a private member variable with a matching accessor function. There’s no easy way around this problem, but thankfully Stéphane Micheloud committed a patch to the Scala trunk to solve this problem. This does mean if you’re planning on using Parcelables, you’re going to need to use a build of Scala that includes this patch.
Scala doesn’t have an equivalent concept to public static final variables in interfaces (and honestly, it’s a bit of a mystery to me why they exist in Java as well). This problem can easily be solved by creating a Java wrapper class that maps values from the interface to member variables on the class. For example,

import com.some.interface.SomeConsts; class Consts { public static final CONST_A = SomeConsts.CONST_A; public static final CONST_B = SomeConsts.CONST_B; ... }

To support anonymous functions and some of the other language constructs that Scala adds, it ends up creating a far larger number of classes than Java. Generally, this shouldn’t be a problem, however, Android appears to have a limit on the number of classes that a single application can have. I don’t have details on exactly what this limit is, but we did find that inheriting from the Scala classes pushed that number up a lot. At one point, we tried inheriting from the Map class and that pushed us over the edge. In general, inheritance can be rewritten as a standard class with the proposed parent class as a member variable, but it’s certainly not as clean.

How Scala?

Before I get too deep into the specifics of how we did it, I want to point everyone to Stéphane Micheloud site on putting Scala on Android. Not only is this site a great resource, but Stéphane was immesurably helpful in getting us up and running.

Our build process consists of three steps, the first compiled the Google Protobufs that we use to communicate between our client and the server, the second invoked the Scala compiler to build the Service and finally we went back to the Java compiler to build the Application level code. I’ve attached the full versions of our build.xml and build-scala.xml (a modified version of the one on Stéphane’s site), but I’ll call out a few things here that might be of interest.

<property name="extensible.classpath" value="${scala-library.jar}" />

If you’re accessing any Scala objects in Java, you’re going to need to let the Java compiler know how those objects work. Unfortunately, there’s no mechanism for extending properties in ant, so when we started working on a new feature that involved Google Maps, we had to change this to:

<property name="extensible.classpath" value="${scala-library.jar};${path.to.maps.jar}" />

<target name="-service-pre-compile"


        description="Compiled the protocol buffers"
        depends="-aidl, -resource-src">
  <javac encoding="ascii" target="1.5" debug="true" extdirs=""
         includes="src/com/google/**,src/com/bump/proto/**"
         destdir="${out.classes.absolute.dir}"
         bootclasspathref="android.target.classpath"
         verbose="${verbose}">
    <src path="." />
    <classpath>
      <fileset dir="${jar.libs.absolute.dir}" includes="*.jar" />
    </classpath>
  </javac>
</target>

This is our compile step for protocol buffers. It looks just like the version in the Android SDK, except we’ve given it a new name and restricted the files it’s compiling with the includes statement.

The is the step to compile the scala code (from build-scala.xml). One thing we changed from Stéphane’s version is we’re using fsc in instead of scalac. fsc (Fast Scala Compiler) acts just like the regular Scala compiler, but it’s faster. It does, however, do some caching to speed things up, so if you’re trying to do a clean build, you need to tell fsc (fsc -reset).

Jamie uses vim and I use emacs, and both of just like the command line too much to use Eclipse, so this is where I’m going to have to stop. Since I know many of you use Eclipse, I’ll call out Indy and Jake to talk about how they handled the Scala code in Eclipse.

Want to work with Will and the rest of the Bump team? We are hiring: http://bu.mp/jobs.

Introducing Stud

Fri, 10 Jun 2011 13:51:00 -0400

This post was written by Jamie from Bump’s server team.

So a few weeks ago, when the zombie army lurched its way toward our servers, a long-neglected scaling issue likewise rose from the dead.

Like all responsible companies, we strive to keep our users’ data protected; so, we tunnel our proprietary socket protocol in TLS. Unfortunately, our use of other-than-HTTP immediately removes the usual candidate for TLS/SSL termination—nginx—from the running.

So, when we were designing our backend systems last summer, we grabbed the only serious open source contender for adding TLS to an arbitrary socket: the venerable stunnel project. Stunnel has three “threading models”: ucontext, threads, and processing.We decided to try each and see what worked best.

We had high hopes for ucontext, since the low memory overhead of setjmp/longjmp-based coroutines would work really well for our particular application. As opposed to something like HTTP, which is typically comprised of many short-lived, active connections, Bump holds open low-bandwidth connections to a very high number of clients at once. So minimizing the memory penalty of each connected client was important to us.

But the ucontext threading module of stunnel performed surprisingly poorly. We could saturate one core with only ~1k mostly-idle concurrent connections—and there was no graceful approach for utilizing more than one core without introducing another load balancing layer. This may very well have been due to some of the flaws the folks at RethinkDB recently documented on their blog

Next, we tried using the pthread-based threaded mode. As programmers, we trust threads about as far as we can throw them—but admit that written skillfully and very carefully, threaded programs can perform excellently. Unfortunately, when put under load, stunnel threw all kinds of nasty assertions and segfaults in threaded mode.

So prefork was looking pretty good at this point. We happened to be using machine with a ridiculous amount of memory (north of 80GB), and copy-on-write semantics meant that each child process would only use around 2MB of resident memory. Furthermore, under the concurrency estimates we had at that time (as well as 12-month projections), the OS scheduler would handle things quite nicely—and to top it off, fork-based multiprocessing is an extremely robust way to run servers, as Apache, PostgreSQL, Unicorn, etc, have proven time and time again.

All in all, we were feeling pretty pleased with ourselves until those pesky zombies *octupled* our concurrent connections overnight.

(If you were wondering, by the way, where process scheduling breaks down—at least, on a mostly-stock Linux kernel—the magic number seems to be around 10-15k processes. By 20k, you are basically doing no actual work—only context switching. And fork() can take upwards of 5 seconds.)

I suppose we *could* have just fired up lots of servers with lots of RAM and spent our way out of the problem. But the whole situation was starting to get a little ridiculous. So we decided to try our hand at a solution.

The result is stud. Stud is the Scalable TLS Unwrapping Daemon (and yes, we worked very hard to make that acronym fit). It takes the nginx model of using one process per CPU core and then doing asynchronous I/O within each process. It has a low memory overhead per secure connection and it minimizes heap allocations to the extreme. It’s easy to use and easy to configure.

Since deploying stud over a month ago, memory usage on our load balancing machines is down tenfold. Load is down 80%, and TLS handshake times are back under 20ms consistently.

Here’s the load on one of our TLS termination servers running stud, handling over 3,000 secure connections (including ~50 handshakes a second):

We decided to open source it in case it helps anyone else out there keep the zombies at bay—it’s a particularly natural partner for haproxy, which is our production deployment. Feel free to drop me a line with any feedback, file bugs on github, or submit pull requests—we’d love improvements!

Want to work with Jamie and the rest of the Bump team? We are hiring: http://bu.mp/jobs.

How to DDOS yourself

Wed, 08 Jun 2011 11:27:00 -0400

This post was written by Michael from Bump’s Android team and is part of a series about lessons learned while developing Bump 2.0 for Android.

One of the key things that any website worth their salt does is collect log data and make the most of that data. This includes things like tracking what pages your users are accessing on your site, and also things like errors that happen or slow response times. The user behavior data can help you build a better product and of course monitoring errors alerts you to problems that need to be fixed. For mobile apps, this is not as easy to do since a lot is happening on a user’s smartphone instead of on a web server in your data center. Many mobile developers used 3rd party analytics services like Flurry, but since we already have a network stack at Bump, we do our own logging. One decision to make is when to send the logs to the server. We don’t want to be constantly using your data connection to send logs to Bump. We especially wouldn’t want to be doing this while you were using Bump. You shouldn’t have to wait for us to send a log file before you can send a picture to your mom.

As part of our 2.0 release of our Android app, we thought we came up with a clever way to get these logs back to our servers without ever bothering our users. You see on Android there are many events on the phone that an application can listen for and thus respond to. In particular, the Android OS will fire such an event when you plug your phone in to charge (a broadcast Intent whose action is android.intent.action.ACTION_POWER_CONNECTED if you are playing at home). Most people plug in their phones at night when they are going to bed, and thus not using their phone. This seemed like a great time to send log files to the Bump servers.

The plan was simple. The Bump app registered a BroadcastReceiver that listened for this power connected event. This caused the Bump background service to wake up and start an upload of log files. Once it finished, the service would go back to sleep. What could go wrong?

There was one obvious (at least in retrospect) thing that we took for granted. We assumed that this event would be fired exactly once when the phone was plugged in and then never again until the user’s phone was unplugged and plugged in again. This assumption turned out to not only be false, but nearly fatal for our servers.

Shortly after Bump 2.0 for Android came out, we started noticing a disturbing trend in our server traffic. We would see large numbers of phone connect to our servers and then do nothing — except for tie up connections to our servers. Of course we first noticed this one day while we were all hard at work here in our office in Mountain View. Further we noticed that most of these “zombies” were coming from Japan and South Korea. We thought that perhaps there was something odd about the phones being sold over there or maybe with the wireless networks.

As the number of downloads of Bump 2.0 continued to climb, things got worse. We had first noticed the zombies from Japan because they showed up during the work day in California. There turned out to be many more zombies right here in the United States.They came out at night, every night, and went away in the morning. We didn’t notice them until they set off monitoring alarms in the middle of the night here. One night we had to block all connections coming from Verizon IP addresses in order to keep the zombies at bay.

This pattern of zombie phones late at night suggested that our send logs to the server strategy was to blame. So we changed that code, and no longer listened for the power connected event at all. This immediately killed off the zombies. Still it left us scratching our heads about the cause of the zombies. So we wrote a little test app. This app would simply count the number of times the power connected event was fired on a device. Then we would plug in a device for awhile and see how many events the app had counted.

We started testing devices, and we began with Verizon phones since we had seen large numbers of zombies on Verizon’s network. Most phones behaved just as they should, firing the event right when the phone was plugged in and never again until you unplugged it and then plugged back in again. But one Verizon phone we tested did not: the Verizon Fascinate. Once plugged in to charge, this phone would fire the power connected event every ten seconds or so (there actually was quite a bit of variance.) With our old log uploading code, this would cause our background service to wake up every ten seconds, shake hands with our servers, and then sit idle. In other words, it would create a zombie.

If you are familiar with Android phones, you will recognize that the Fascinate is Verizon’s version of the Samsung Galaxy S. The Galaxy S is perhaps the most popular Android phone in the world. We tested several other Galaxy S phones, and none of them had the zombie gene like the Fascinate. The presence of zombies in Japan and South Korea suggests there are phones being sold there that have the zombie gene, perhaps some other flavor of the Galaxy S. We can’t say which one for sure. We’re just glad that we can once again sleep through the night without being woken up by zombies.

Want to work with Michael and the rest of the Bump team? We are hiring: http://bu.mp/jobs.

iOS vs. Android from the Trenches

Wed, 18 May 2011 15:09:00 -0400

This post was written by Indy from Bump’s Android team and is the first part of a series about lessons learned while developing Bump 2.0 for Android.

After months of hard work, we shipped Bump 2.0 for Android at the end of April. It was a full rewrite of the client from the ground up to bring it onto Bump’s 2.0 architecture. To make this happen a couple of us at Bump switched to working full time on Android from iOS, and experienced all the joys and pains that come from such an experience. We’ve teased our former iOS teammates about how certain things are so amazing and easy to do in Android, but the truth is more nuanced. In this post I want to spend some time looking at the similarities and differences of the two platforms from both a technical perspective and a UX perspective. Though it is possible to separate these two, since we are talking about a handheld device, both these perspectives are often intertwined.

For 2.0 we wanted to build an App that felt native to Android and not a shoddy copy of an iPhone app. We ourselves are guilty of this with the previous version of our Android app (Android doesn’t have tabs on the bottom!). We went through several iterations of our initial specs for 2.0 to bring it closer to a native Android look and feel.

Aside from the surface level UI differences, I found that Android’s architecture is very forward looking in comparison to iOS. Every app gets a chance to deeply integrate with the operating system and other applications. There is a lot that the OS has to offer in this respect and so far we’ve only scratched the surface of what we can do with this at Bump.

One example of a future looking architecture decision is how different applications can flow back and forth between each other. There are two major parts of the system that make this possible: Intents and the hard back button.

Intents are part of the notification system that is built right into the operating system. Any application can listen to and act upon intents that are broadcasted system wide. For example: when a user clicks on a link inside a chat in Bump, an intent to view the link is broadcasted by the system. The browser is registered to respond to it so the system slides the chat screen away and slides in the browser with the contents of the link. The user now has the power of the full Android browser to interact with the web page. When the user is done viewing this link hitting the back button slides the chat screen back in.

This is a huge UX win! In iOS there is simply no way to make this sort of seamless interaction. Apps have to bake in a UIWebView and then give you an option to open the page in Safari if you wanted get all the browser features. Once there getting back to the previous app is a multi step process. Android is filled with many such integration points that help users flow seamlessly between tasks. As an app developer you can even add new integration points for other apps to interact with yours.

When it comes to layouts, Android has to deal with many different screen sizes and resolutions. It provides many tools to deal with this variation. One such tool is declarative (in xml) layouts that “flow”. These layouts are very similar to how layouts flow on the web. Each object in the layout has its position either defined based on an edge of the screen or relative to other objects. This layout methodology allowed us to quickly iterate on the UI by simply moving bits of xml around. Also simple things like a line of prompt text that might become two in more verbose languages (*cough* German) don’t break the UI. On screen elements simply shift to adapt to this change.

There is, however, a flip-side to flow layouts. In iOS you can often do pixel perfect placement. In Android with it’s multitudes of resolutions and screen sizes you simply don’t have that luxury. Thus the same tricks for placing elements in iOS simply don’t carry over. However, this is something every designer that has moved from print to the web has had to deal with, and it’s not a new problem.

Android also allows an app to run multiple OS level processes. We take advantage of this ability to do almost all of our CPU heavy and networking work in a separate background process. This is great because we don’t have to worry about any of this work having much of an effect on UI performance. This is a constant struggle in iOS even if you manage your threads well.

One thing that has been a disappointing with Android though, is that animations and responsiveness of the UI is generally slower. In the parts of our app that we animate, we often see missed frames. Honestly this sucks. There are definitely things that we can do to make this better but in the end this is because none of the graphics and animations are hardware accelerated pre Honeycomb. Unless you specifically write Open GL code (and now Renderscript) you won’t get the same level of graphics performance that you see in normal iOS apps. Since currently only tablets run Honeycomb, we are stuck with generally not so stellar graphics performance on pretty much all current handsets.

The list of differences between the two platforms is obviously enormous. There are quite a few purely technical differences, but discussing many of them can easily degrade into religious arguments (languages, dev tools, garbage collection vs. ref counted memory management). In most of these regards both platforms are fairly comparable. The major difference being that Android’s Java code runs on a relatively young (dalvik) VM that hasn’t figured out all of its optimizations and bugs yet. It’s quickly catching up to its older VM cousins though.

Building for Android was definitely quite a change from iOS, but both involve building for devices with very constrained resources. We learned a lot and had some very interesting experiences. In the coming weeks we’ll share more of these on this blog, including something we’ve been asked often about: using Scala in Android.

Want to work with Indy and the rest of the Bump team? We are hiring: http://bu.mp/jobs.

TL;DR.js

Wed, 11 May 2011 14:54:30 -0400

Reading all the hacker news articles can take too long.

Sometimes an author will add a tl;dr (too long; didn’t read) section at the bottom for people who just scrolled by the article without reading it. For those authors who weren’t so considerate, here is a script that will do it for you. You install it by dragging it to your bookmark bar, then when you accidentally click on a really long article, don’t worry, you don’t have to read it. Just click the link in your bookmark bar and a nice little overview will show up at the top of the page. YMMV.

Install by dragging this link to your bookmark bar (in chrome click view->always show bookmarks bar to see it):

If you want to summarize a subset of the article, select it with your mouse then click the bookmarklet.

It works by finding the most common words in the article, and then finding the sentences with the highest density of common words (excluding words like “the”, “and”, etc.)

Here is the tldr of Daring Fireball:

Apple un-magic is interesting

“Open” is to Android as “magic” is to iOS

iOS aims for it Android doesn’t want it ★

Google hypocrisy is interesting; Apple secrecy, not so much

Android feels like an independent Google subsidiary

This is a quick hack so please fork on github:

tl;dr.js on GitHub

Spend less time reading hacker news, join Bump: http://bu.mp.jobs

So you want to do Android development at Bump...

Fri, 15 Apr 2011 18:10:00 -0400

This post was written by Michael from Bump’s Android team.

So you want to do Android development at Bump…

Well of course you do! However, Android is a hot technology, and there are a lot of companies trying to hire Android engineers these days. Here at Bump we have a little different approach to hiring for our Android team than other companies. Hereʼs an example of an Android job listing pulled from Dice.com and the Android developer position here at Bump:

The Other Guys

Hands on development experience with Android (2 plus years)
Java programming (8 plus years)
Expert knowledge of circuit boards, processors, electronic equipment and computer hardware and software
Expert knowledge of Oracle, C#, .NET
10+ years experience required
Android SDK, Java, iOS, Agile, Java MBeans Extensions, JMS, javascript

Bump Technologies

Experience developing native Android apps or other mobile apps is strongly preferred
Java; an interest in learning Scala (or experience)
An eye for great design and user experience
A desire to create one of the best apps in the Android Market
Ability to communicate within a close-knit team

I found this listing to be pretty humorous. The Android SDK went 1.0 at the end of 2008. So to have 2+ years of Android experience means that any potential candidates for the above job would have had to have jumped on the Android bandwagon back when the T-Mobile G1 was the only phone in the world running Android. So this company is looking for an elite early adopter, who also happens to be an expert with circuit boards and Oracle (database?) Compare this to what we are looking for at Bump. You might argue that our standards are lower. Certainly we have less requirements, and our requirements are not as speciﬁc. However I would argue that our standards are actually much higher.

Android is such a new technology that requiring tons of experience is quite limiting. Iʼd go as far as to say it is foolish. Good engineers can learn a new technology much easier than an experienced, but average engineer can suddenly take their game to the next level. Asking a talented engineer to learn a new technology is exactly the kind of risk that startups have to take, and it is deﬁnitely a given here at Bump. As an aside, I would conjecture that this hiring principle is a key reason why startups are able to innovate much faster than bigger, badder companies. Certainly at Bump weʼre not looking for another cog in the machine, weʼre looking for people that can raise our IQ and enhance our DNA.

Ok enough with the propaganda, letʼs get a little more concrete about what it takes to be an Android engineer at Bump. It should be obvious by now that we are not requiring Android experience. The ﬁrst most important thing is that you are a great engineer. Of course everyone who applies for an engineering job in Silicon Valley is smart and gets things done, so that is not enough. At Bump our focus is on letting folks use their smartphones to interact with the physical world. So we routinely ﬁnd ourselves with very unique problems to solve — so even if you have tons of experience, it may not give you much of an advantage in solving some of the problems we face. Thus we look for some knowledge of software engineering/computer science “theory”. If youʼve ever interviewed for a programming job in the Valley, you probably know the drill. Youʼll be given some problems to solve on a white board, and they will probably involve some essential algorithms, data structures, etc.

Programming on a white board can provide some useful insights in to how a developer thinks and what kind of theoretical knowledge they bring to the table. However it is a far from perfect way to evaluate programmers. It really has little in common with how most programmers actually do their day-to-day job. So before you ever step into our ofﬁce and pick up a dry-erase marker, we want to see something real: real code or a real app that you are responsible for. Now I know what youʼre thinking. If youʼre being asked to send code in to a company, chances are you can come up with a clever enough search on a decent search engine to ﬁnd all or most of the answer. And then thereʼs always Stack Overﬂow, but I digress. The real coding problem is a good ﬁlter. It also gives us a lot of insight into what you think is important when you code. Is the code optimized for speed or memory? Does it have a procedural, object-oriented, or perhaps even a functional style to it? Do you have an eye for usability? Combine this with some white board pseudocode and we have a better shot at evaluating a candidate.

Programming problems at home and on white boards provide an objective way to evaluate an engineer. Pure programming skills may be a necessary condition for working at Bump, but it is not sufﬁcient. There are a lot of softer things involved too. One of those is passion for technology. For an Android engineer that should mean passion for Android (if you walk in to an interview sporting an iPhone … well Iʼll leave it at that.) So even if you do not have Android experience at your most recent job, we still expect that Android is at least a serious hobby of yours. Thatʼs why you should expect the at-home programming problem we give you to involve writing a working Android application. We want both code and an APK! We want people who want to develop for the Android platform, not people who just want to collect badges for their resume. So there you have it. Hopefully after reading the above youʼre now dying to see what kind of Android programs weʼll ask you to write. Thatʼs the kind of passion that weʼre looking for, and we think itʼs much more important than how many years of Android experience you have. If you think that describes you, then apply now.

Safeguarding ourselves from ourselves

Wed, 13 Apr 2011 16:47:00 -0400

This post was written by Seth from Bump’s iOS team.

Managing a large code base can be a challenging task, especially as a team grows and more people are contributing to different areas at the same time. In iOS land, there are a couple gotchyas that seem to crop up from time to time and can really bite you. Once bitten, twice shy. Whenever we run into a problem, first we figure out what went wrong, next we educate the team about the root problem, and lastly we try to protect ourselves. Our frail human memory has a habit of forgetting little details that are important, and every time we add someone new to our team it’s not practical to review every little thing we’ve learned so far. We try to create safeguards that will protect ourselves from… ourselves, tools that help us catch the mistakes that slip in now and again.

One such tricky problem is UIViews and background threads. Apple’s UI code is not thread safe: you can’t go adding and removing views from multiple threads at the same time without the risk of corrupting your views and possibly crashing. In general we do a good job at not doing these terrible things when not on the main UI thread. However, we’ve occasionally had little bits of code that, either directly, or indirectly, cause some view code to be called in the background. The end result is indeterminate race conditions and strange crashing from autorelease pools.

Here’s one tool we use to protect ourselves from this: a UIView category that ensures you do certain things like autorelease, and release on the main thread, things which can be subtle and hard to detect, yet potentially discrashterous. It throws an exception while debugging so that you can see which piece of code in your call stack is offending. Of course, we can only defend against mistakes we’ve already made, but at least we won’t make them again. And maybe it will help you too.

Are there things that have bitten you once, that you’ve coded against?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53

// UIView+Threading.m

// Created by Seth Raphael on 4/12/11.

#ifdef DEBUG

@implementation UIView (UIView_Threading)

- (id) autorelease;

{

if (![[NSThread currentThread] isMainThread]) {

[NSException raise:@”UIView Background Exception” format:@”Autoreleasing a UIView %@ on a background thread”, self];

}

return [super autorelease];

}

- (void) release;

{

if (![[NSThread currentThread] isMainThread]) {

[NSException raise:@”UIView Background Exception” format:@”Releasing a UIView %@ on a background thread”, self];

}

[super release];

}

@end

@implementation UIViewController (UIViewController_Threading)

- (id) autorelease;

{

if (![[NSThread currentThread] isMainThread]) {

[NSException raise:@”UIView Background Exception” format:@”Autoreleasing a UIView %@ on a background thread”, self];

}

return [super autorelease];

}

- (void) release;

{

if (![[NSThread currentThread] isMainThread]) {

[NSException raise:@”UIView Background Exception” format:@”Releasing a UIView %@ on a background thread”, self];

}

[super release];

}

@end

#endif

Want to work with Seth and the rest of the Bump team? We are hiring: http://bu.mp/jobs.

How we use Redis at Bump

Tue, 15 Feb 2011 13:14:00 -0500

This post was written by Will from Bump’s server team.

We’ve been using Redis at Bump for a number of things since we launched our 2.0 platform last July. After reading Francesco’s recent post about Redis I was inspired to enumerate the many different roles Redis plays here. I also stuck in a few best practices that we’ve learned along the way at the end.

Message Passing
We originally looked into using AMQP for communicating between multiple servers and processes, and were frustrated with the complexity of it—we’re not a bank nor do we want to deal with the overhead required for complex transactional messages. Enter Redis lists. The combination of lpush and brpop create simple queuing and message passing behavior. BSON encoding the data we send over the wire provides a simple translation between python dictionaries and a reasonably efficient wire format. (Note: I use queue and list here to represent the user-level functionality being provided, however, to Redis, they are all just lists.)

Adding an rpc mechanism simply requires sending a header that defines a callback queue to listen on (this can be generated at a uuid4 or anything else suitable collision free) along with the original message. The sender calls lpush, followed by brpop on the callback queue and the receiver, whenever it’s ready to reply, lpushes it’s response onto the callback queue. Since there’s nothing left in the callback queue, Redis takes care of cleaning it up for you.

Logging
We run a number of different processes on a number of different servers to keep Bump up and running. Instead of trying to keep track of text logs spread across multiple machines we just push any log line we are interested in onto a single Redis queue. Then, we have one process pop from that queue and write those log lines to disk. If that process or the logging server goes down, no big deal, events just queue up in Redis until the logging service is back online.

Social Graphs
We recently released the ability to make Bump friends without actually bumping and with that came the problem of constructing and storing a social graph of our users. Already knowing Redis well (and knowing it was fast), we turned to Redis sets. Want to know what friends I have in common with Jamie? We keep all of Jamie’s friends and all of my friends in separate sets in Redis, so the friends we have in common is just a set intersection, a function Redis already provides on sets.

Asset Caching
Like most companies these days, we’ve gotten out of the storage business, leaving the heavy lifting to Amazon. Unfortunately, the latency to S3 wasn’t as low as we wanted for our most active assets. At this point I imagine you’ve guessed the trend, but Redis has a solution for this too—max memory and lru mode. Redis 2.2 (just recently released) can be run in a mode where you specify the maximum amount of memory Redis can use and also specify the record eviction criteria. When an asset gets set up through Bump we put in into the Redis cache and eagerly store it to S3 as well. The fetch of an asset simply checks Redis, if it’s not there, repopulates the cache from S3 and tries the fetch again. LRU mode takes care of the eviction of items that haven’t been used in a while. Perfect.

We have ended up using Redis for a host of other smaller things as we continue to grow. Although this wasn’t an explicit decision, Redis provides enough basic data types and functionality to get most problems done and you’d be hard pressed to find something that can do it faster.

That said, we’ve run into a few issues along the way. None of these problems dampen my enthusiasm for Redis, they are worth being aware of, especially if you are planning on using Redis for latency-dependent operations.

Persistence
This is something that every database has trouble with and Redis is no exception. Antirez is hard at work on a new disk-based back-end for Redis that looks like it has promise, but for current production deployments there are two options: periodic background saves and writing to an append only file (AOF). Since periodic background saves can result in significant data loss if the server crashes, that wasn’t a great option, so we settled on using the append only file. This works without a hitch until you need to compact the AOF. Due to the behavior of fsync and the fact that Redis uses a single event loop, at some point in the process of compacting the AOF the main event loop can become blocked on disk I/O. Since we use Redis for time sensitive things like message passing, we can’t afford anything that will hang the entire server up. Unfortunately, there’s no silver bullet here—you just can’t persist an active production instance of Redis. We can’t live without persistence, so we just have the slaves run in AOF persistence mode. The momentary hiccup in the slave just leads to a minor backup of the replication process, not ideal, but certainly not a terrible solution to a hard problem.

Running with Mongo
Very early in our deployment we had Redis and Mongo deployed on the same machine. The whole story here will have to wait for a post on Mongo, but the take away is definitely that Mongo should always be run on a server by itself. That said, it’s worth taking a minute to think about the implications of the single-threaded model that Redis employs. If, for example, malloc takes a few hundred ms to return (the problem we were seeing with instance shared with Mongo) or if something is doing heavy disk I/O and a writing a log line ends up being disk-bound, the entire server will stop responding. Depending on your use case, this can be acceptable or not, but it’s something to keep in mind.

If you’re looking at a new problem and it involves a data type that Redis supports (and really, what problems can’t be solved with lists, sets and hashes) I strongly recommend giving Redis a long look. It’s become the Swiss Army knife here a Bump, except that it opens the bottle of wine faster than anything else, instead of leaving half the cork in there.

Want to work with Will and the rest of the Bump team? We are hiring: http://bu.mp/jobs.

Haskell at Bump

Mon, 24 Jan 2011 17:28:00 -0500

This post was written by Jamie from Bump’s server team.

At Bump, we’re always willing to try something “off the beaten path” to get an edge. For example, our backend systems are almost entirely non-blocking and coroutine based, written in Python using Diesel; we have an in-house analyzer written in Ruby that verifies retained properties are released in our iPhone code; and, when we recently launched our Bump Android 2.0 project, we elected to write large parts of the core code in Scala.

Our use of Haskell, then, fits right in with our “try anything” mantra. Haskell is a lazy, pure (with explicit non-purity), statically typed functional programming language. Its closest cousin is that other family of Hindley-Milner type-inferred languages, the ML-derived SML and OCaml.

While Haskell has long enjoyed heavy use among the CS academic crowd, we at Bump believe it hits a sweet spot of safety, correctness, and speed, making it a great systems language. With the advent of some very serious web frameworks and high performance servers, it appears we’re not alone in this belief.

Our first foray into Haskell was Angel, a daemon monitoring and management system ala daemontools or god. After a very sucessful and stable 6+ months of Angel in production, we were encouraged to revisit Haskell in a future project.

Enter our logging and analysis infrastructure. At Bump, we use Redis extensively as a “swiss army knife” for data; we have over 700GB of RAM dedicated to Redis storage! One of our heaviest uses of Redis is as a messaging/queue server for our centralized logging system. Every single event generated by any backend or frontend flows through redis lists to a central recording/processing system.

After our big christmas, our load has reached the point where, on peak hours, that event stream is ~1500 events per second. Knowing that we had culled all the low-hanging optimization fruit out of the overtaxed Python-based daemon that had been consuming this event queue, we decided it was time to revisit this system altogether. It seemed as good a time as any to give Haskell another spin.

First, we made some modifications to the redis-haskell binding project. While keeping the command layer mostly the same, we swapped out the internals to use attoparsec for protocol parsing and the binary package for protocol generation. After these modifications, operations with Redis were 10x faster than our pure Python equivalent.

Once our work on the Redis bindings was done, rewriting the simple consume-and-log daemon in Haskell took about a day. The resulting daemon is 4x faster than the Python equivalent, and has approximately the same lines of code. Best of all, thanks to a very powerful type system and rigorous control on side effects and concurrency, we only discovered a single bug after compilation.

But there’s still quite a bit of tuning we can do in the future to make it go faster still. First, one part of our process involves translating the event representation from BSON to JSON. Our naive use of various Haskell libraries for this task isn’t very disciplined about avoiding conversions between data representations and string types (CompactString, Lazy/Strict ByteStrings, etc). A custom, minimalist JSON generator tailored specifically to the BSON library’s output could be an interesting enhancement.

In addition, we’re taking very little advantage of Haskell’s very sophisticated parallelism capabilities. We could explore an approach distributing our (de)serializaiton jobs across many cores, using MVars or even full-blown STM to communicate between many forkIO microthreads and a single log writer doing the disk I/O.

Technology changes so quickly, at Bump we believe it’s best to have many arrows in the quiver, and just hire very smart, productive, generalist developers that can do amazing things with any of them. Thus far, Haskell has proven to be a very valuable addition to our arsenal.

Want to work with Jamie and the rest of the Bump team? We are hiring: http://bu.mp/jobs.

Generators and Back Again

Mon, 09 Aug 2010 12:00:00 -0400

This post was written by Jamie from Bump’s server team and is cross-posted from his blog. The code snippets in this entry are all Python-esque pseudocode, unless otherwise noted.

I’ve long held an interest in asynchronous I/O frameworks for Python.

I started out playing around with asyncore/asynchat circa 2001. I spent some time in Twisted, and made some small contributions to that community. Then, I started tinkering with a few micro-frameworks of my own. At my last job, I wrote a slightly-more-complete framework called “eventful” to build a number of tools we ran server side, like a key/value database system and a deployment system.

Over the years, I’ve realized a lot of the evolution the various frameworks underwent was devoted to—somehow—achieve a more “natural” flow than callback style typically permits.

Allow me to explain.

Typically, your code is invoked by an asynchronous framework when some data is available to process:

def handle_read(self, data):
    # do stuff with data

Then, some data-dependent dispatching happens:

def handle_read(self, data):
    if is_put_command(data):
        self.handle_put(data)
    else:
        self.handle_get(data)

And who knows, the framework might provide tools to facilitate dispatching at this level.

Now, let’s concentrate on our get event handler:

def handle_get(self, data):
    resource = parse_get_stuff(data)
    return get_resource(resource)

Note that in our fictional framework here, the handler returns the data to be written back onto the connected socket. So far, so good.

But now, let’s pretend that get_resource(..) involves a call to a database or something. Since we’re in an async I/O framework, everything is running in a single thread. Our handler cannot block, or the main I/O loop will not be scheduled to handle events related to other sockets. So, get_resource itself must be asynchronous. That means it needs the event loop to do I/O on its behalf—so, we need our handler to return some request that indicates more stuff needs to be done by the event loop, then the results of that operation need to be passed back into some continuation to finally arrive at the real “return value” of the handle_get handler.

Callbacks are the way this is classically achieved:

def handle_get(self, data):
    resource = parse_get_stuff(data)
    io_request = get_resource(resource)
 
    def transform_database_data(db_result):
        return db_result['user_id']
    io_request.add_callback(transform_database_data)
 
    return io_request

But this kinda sucks for a few reasons:

The logic appears out of order. The final transformation of the data appears in code before the “first stage” of the handler is actually finished
Traditional exception handling isn’t possible. You need to add an “error callback” handler to every io_request in case an exception occurs in the operation the event loop does on your behalf
You either need to pass through any data calculated at early stages of the callback chain or rely on closure scoping, which works but gets pretty ugly when your callback chain gets longer than a few links

But really, the most decisive reason this is ugly is that people don’t usually code this way.

I think, lacking better metrics, a decent way to evaluate the elegance of a given approach in a particular problem domain is: “if the constraints of this particular domain were removed, would people still code it this way?”

Ask yourself, would you ever write this function?

def print_in_uppercase(phrase):
    def print_it(up):
        print up
 
    op_request = do_for_me(phrase.upper)
    op_request.callback(print_it)
    return op_request

The Solution

Let’s be honest. This is what we want:

def handle_get(self, data):
    resource = parse_get_stuff(data)
    db_result = get_resource(resource)
    return db_result['user_id']

The challenge is get_resource, we need the event loop to take control of the thread again. But the event loop is “up the stack”—it called you!

A first answer might be “well, let’s just call back to it!”. In “normal” Python (and most other languages), however, we have at least two problems with that:

Every time we call over to the event loop, we’re making the stack deeper. We’ll stack overflow eventually.
There is no good way to “resume” our handler when the I/O is done. Heck, the stack might have other frames on it related to other sockets—so we can’t just have the event loop “return” data back up to our handle_get frame.

Generators to the Rescue?

Python async I/O developers’ collective ears perked up when generators were introduced in Python 2.2. When a generator yields, it sort of freezes the stack frame and lets the caller do other work—including scheduling other generators. Then, when .next() is called on that generator, the frame is resumed right where it left off. This allows for cooperative multitasking patterns, sometimes called coroutines, like this (this is runnable code, not pseudocode):

def one_g():
    for x in xrange(3):
        print "one!"
        yield 
 
def two_g():
    while True:
        print "two!"
        yield
 
def scheduler(loops):
    for x in xrange(3):
        for l in loops:
            l.next()
scheduler([one_g(), two_g()])

Output:

one!
two!
one!
two!
one!
two!

All that was missing was the ability to pass something back into a generator; then, you could have one generator affect the behavior of another.

Fortunately, that was added in Python 2.5 when generators grew a .send() method that took an argument that would be the return value of the last yield statement. So, now we can start to put something together that acheives our handle_get goals:

def handle_get(data):
    resource = parse_get_stuff(data)
    db_result = yield get_resource(resource) # returns control to the scheduler
    yield response(db_result['user_id'])
 
def do_db_work(request):
    # io stuff
 
def scheduler(loops):
    next_get = wait_for_get()
    getter = handle_get(next_get)
    val = getter.next()
    if val is get_resource:
        result = do_db_work(val)
        val = getter.send(result)
    assert val is response
    send_to_socket(val)

The rough idea is, the framework does the heavy lifting to make generators look like near-seamless coroutines to application code.

Diesel is Born

In Summer 2009 I was building a web app with a couple of friends. We were doing all sorts of Comet-y and event-y stuff, and I decided to gather up my decade of callback scars, my Python 2.5 generators, and take a stab at putting together something cool.

So we started the Diesel project. Take a look at the docs to see lots of examples—but in a nutshell, it let you do very cool stuff, written in an intuitive, blocking style, with minimal code.

Here’s a basic echo server, for example:

def echo(remote_addr):
    their_message = yield until_eol()
    yield "you said: %s\r\n" % their_message.strip()

Diesel managed huge bunches of these generators, and looked for specific tokens, like until_eol, which gave it (ie., the event loop) instructions on what kind of I/O to conduct, or how long to wait to resume the loop, or what “event” to wait on.

And it worked pretty well… but it wasn’t all wine and roses.

Degenerate Generators

As we’ve long known: the world is not flat.

For programmers, the world is nested. And these nestings are the building blocks for generality, encapsulation, etc. Functions and objects and methods allow us to adhere to the policy of “Don’t Repeat Yourself”.

But these constructs are problematic for something like Diesel:

def get_user(uname):
    do()
    several()
    io()
    things()
 
def handler(username):
    if username is not None:
        user = get_user(username)
    else:
        user = get_user('default')
 
    yield user.to_string()

The fundamental problem is Diesel must be the “caller” of the generator—the parent frame—in order to process tokens destined for it.

The call pattern we all know and love:

Diesel calls handler
handler calls get_user
get_user returns to handler
handler returns to Diesel

…will not work! If get_user does any IO, Diesel needs to be the recipient of the tokens being yielded to the caller. Once again, the issue is the event loop needs to be in control.

This puts Diesel in the business of “faking the call stack”. This convoluted mess is what’s required:

Diesel calls handler
handler yields another generator (get_user) to diesel
diesel pushes get_user onto the stack and calls .next() on it
get_user does IO, yielding signals to Diesel
eventually, get_user yields a "return value"
Diesel pops get_user off the stack 
Diesel calls .send(..) on handler, passing in the return value from get_user
handler resumes with the result of get_user
eventually, handler yields a "return value"

Oy! Where do I begin?

This code is ugly and hard to get right. It’s slow (compared to native function calls, which are relatively slow in Python already). Exception handling being raised “up the stack” (of generators) is near impossible to do right and imposes un-pythonic things on the application code. A funky token needs to be yielded that approximates the behavior of the keyword “return”—push something up to the caller in the generator stack and end the frame’s execution. And it’s very easy to make mistakes in your application about when you’re yielding generators vs. calling regular functions, and hard to refactor one into the other.

The Greenlet Kool-Aid

We’ve been busy at Bump writing new backend systems that power version 2.0 of our iPhone app. We had 10k+ lines written on Diesel and generators, and things were mostly working… but we were fatigued of running into the aforementioned complications. We wanted regular function calls, regular tracebacks, and regular exception handling.

So I started investigating using the greenlet library as an alternative to generators.

The greenlet library uses some assembly trickery to do true coroutines in Python. This means you can “call over” into another user-space microthread without going deeper in the stack, and then resume the caller later on.

Here’s an example from the greenlet website:

from py.magic import greenlet
 
def test1():
    print 12
    gr2.switch()
    print 34
 
def test2():
    print 56
    gr1.switch()
    print 78
 
gr1 = greenlet(test1)
gr2 = greenlet(test2)
gr1.switch()

That switch() call is the magic—jumping between arbitrary call stacks at will!

Diesel 2

One furious night of coding later, I had a branch of Diesel ported to use greenlet at its core instead of generators. In the process, I removed hundreds of lines of code full of edge cases, and the result was a codebase with fewer bugs (even though it was younger) that was much faster to boot.

We’ve been running millions of users on Diesel 2 at bump for the last few weeks since the launch of the Bump 2.0 app, and everything is going along swimmingly.

Here’s our contrived application, Diesel 2 style (using MongoDB):

def get_user(uname):
    client = MongoClient()
    return client.myapp.users.find({'username' : uname}).one()
 
def handler(username):
    if username is not None:
        user = get_user(username)
    else:
        user = get_user('default')
 
    return user

I feel somewhat conflicted, because the generator approach, with yields aplenty, was the a key characteristic of Diesel when we launched the project; but I eventually had to concede that the decision to use the greenlet library instead of generators for the basis of Diesel was, practically, the right call.

Try out Diesel 2: GitHub

Want to work with Jamie and the rest of the Bump team? We are hiring: http://bu.mp/jobs.