Creating Super Feeds
5 minute read |Cloud
My goal is to help everyone better understand security and privacy as it relates to technology. In order to deliver on that goal, I need be learning constantly. The good news? I love learning new things. The bad news? There’s a massive amount of new information being published all of the time.
The technology community (both building and using it) is a very active one. There’s always new ideas to explore and discuss.
However there’s a danger there too. It’s way too easy to get lost down the rabbit hole bouncing from site to site.
I need to be able to quickly consume the latest information from a wide variety of sites.
This is not a new problem.
In fact, way back in 1997, Dave Winer created a way to easily get the latest posts from his site. He called is RSS or really simple syndication. The idea was truly simple. Each site would publish a specifically formatted page (called an RSS feed) that would like the latest updated for that site.
Viewer could use then quickly see the changes.
Nothing is truly simple and Dave’s format was one of several vying for popularity. There were even others called RSS creating a number of problems for readers who just wanted updates from their favourite sites.
At least everyone agreed that this was a solid idea. RSS, Atom, CDF, and others quickly faded to the back as reader could simply subscribe to a sites “feed” and get updates either in directly in their browser or via a dedicated newsreader app.
The most popular of these newsreader apps was called Google Reader. Launched in 2005, it quickly rose in popularity until it trailed only GMail. Google Reader made it easy to quickly add feeds from your favourite sites so you could quickly read through them in one place.
I can’t state how amazingly useful this was. To this day, it still stands as one of Google’s best products ever. A lot of people (myself included) are still bitter that it joined the long list of services killed off by the company when it shuttered in 2013.
Thankfully newsreaders still exist today. Feedly (launched in 2008) retains much of the same utility but still doesn’t echo that same unique feel of Google Reader. It is just as useful but never quite achieved the same popularity.
On the desktop, apps like Reeder and others continue to offer feed subscription and consumption capabilities.
In order to support Feedly, Reeder, and other newsreader apps, a list of feeds format was developed in 2000. An OPML document was simply an XML formatted document that contained the feeds you were interested in and optional groupings.
OPML didn’t really take off as it’s primary use was to allow people to quickly and easily configure their feed preferences.
It’s still supported today by the vast majority of newsreaders but it turned out to simply be not as useful as initially thought.
In fact, RSS feeds in general are slowly disappearing from the web. More and more commercial publications are more worried about owning an audience vs. building one. Social media has replaced tailored consumption.
It’s an understandable shift but also a bit of a sad one.
RSS feeds will never disappear completely as they power the podcast world. When you “subscribe” to a podcast, you’re actually just configuring your podcast reader to check the shows RSS feed on a regular basis.
In fact, Apple Podcasts is simply a giant RSS feed directory!
There is life left in the format still and it can definitely help solve my challenge around continuous learning…hopefully.
I started combing through sites that I regularly visit for information on cloud, DevOps, and security topics. I found 252 (to start with) that had active RSS feeds. Excited, I started to add them to Mailbrew.
Mailbrew is a simple service that I use to email me the latest information on various topics each day. I find that getting these updates via email fits my work flow better than a dedicated reader app. I can’t avoid email!
…ok, I can and I apologize to everyone that I haven’t responded to lately. Let’s re-phrase; it’s harder to avoid in an email!
The problem is that I would have to add each of these RSS feeds to Mailbrew one-by-one. That was going to take way too long.
💡 Why not create a “super feed” that contained all of these sources?
Coding For Fun
- super feed
- my solution
- aws lamba vs. fargate
- architecture (+ gist)
This is not a new problem. Way back in 1997, Dave Winer, created the RSS—or Really Simple Syndication—format. The idea was simple; each site would publish an RSS document (know as a feed) that would list the latest pages updated on the site. Viewers could download that file and see what was new.
Nothing is every truly simple and it turned out that this was only one of several formats—some also named RSS!—that were competing to be the standard.
At least the ideas were consistent at a high level. A site updates a document that viewers can subscribe to (regularly download) in order to stay up to date.
These technologies came to mainstream prominence via Google Reader. By far the best product Google has ever killed off. This tool was launched in 2005 and let users add a series of RSS feeds. Google Reader would then show all of the content from those feeds in a single, easy to use interface.
Over Google Reader’s 8 year run, it was a shining example of the open web.
Anyone can publish an RSS feed. It’s just a specifically formatted XML document. Most web platforms offer an RSS feed by default.