Machine Learning has gradually trickled down its capabilities to every little inch of the user experience. The kind of actions you would have normally have to do yourself are now being delivered at your fingertips before you even need them. Twitter is trying to automate another one of these processes with the help of machine learning — thumbnails.
Yes, the post previews you wouldn’t think needed any sort of advancement. Generally, thumbnails are essentially cropped versions of the original image and in most cases, you end up enlarging it manually for knowing what’s it about. That’s primarily because the cropped picture doesn’t necessarily always offer a quick glance over its piece de resistance as the algorithm behind it is largely functioning randomly or invariably trims a specific portion.
With Machine Learning, however, Twitter will now process each image and try to figure out the most interesting part of any picture you post. To achieve this, the engineers trained a neural network by inputting data from various studies which researched the areas of any given image people look at first.
But hey, real-time machine learning costs a ton of processing power. So how do you maintain a lag-free user experience and implement these new demanding algorithms? The answer lies in the large size of thumbnails. You see, Twitter doesn’t need to get in the details, it merely wants a rough estimate of an image’s most salient features and crop it down to that level. The technique, if you’re interested, is called “knowledge distillation” and it allowed Twitter to speed things up by a significant percent (10x).
Incidentally, Twitter has tried this approach before but they forgot to consider that the pictures posted on a social network are not just about humans. There are scenery, objects, pets and a whole lot more. So for the past few months, they’ve been busy extending the algorithm to be compatible with information beyond faces. The new feature is now rolling out to their smartphone apps and the desktop website for all users.
I guess this is where those “Open for a surprise” memes and Twitter part ways.