The good old days
LAMP strikes a nostalgic bell to me. It stands for Linux – Apache – MySQL – PHP, and I first heard of it in 2005, when it was pretty popular. Having these four great technologies around for free at that time meant that any kid with a computer and an Internet connection could host his own website by investing nothing more than a little time into it. Together, they solved the issue of infrastructure for most of the time’s use cases, and they encouraged innovation, in what I think was a truly impactful time in software history. Open-source was flourishing, and it was becoming annoying enough to large enterprise players to trigger memorable remarks like “Linux is an OS fit for communists“.
All of us in the Telepat team have some sort of memory related to this stack. LAMP was awesome, and it taught us and the world about the benefits of free software.
Fast-forward to today, when it is “almost impossible to find a member of the Fortune 500 who is not using Node in some capacity”, we can get a pretty clear picture of how much open source has influenced the world around us. It’s now touching most of the apps we interact with every day on our mobile phones, most of the websites we use and most content we consume, one way or another. And, unlike ten years ago, the big companies are leading the way, with the likes of Facebook, Microsoft or Adobe releasing large catalogues of open source components to the public. Not even Apple or NASA can stay away nowadays.
So what happened meanwhile?
Code got neatly packaged. There are more than 150k code packages available on npm alone, so whatever functionality you need, there’s probably a package for that. We use (a lot of) package managers to efficiently ingest code in our projects, by tapping into the vast, open source pool that GitHub made available, On-demand code – after going through times when even tasks like creating a crypto hash or sending an HTTP request meant reinventing the wheel, that’s amazing.
We also have APIs now. They rose from a need to share business data across web applications, something that the traditional HTML model didn’t do so well, and have since sparked a data revolution. APIs are now the invisible pipes of the Internet, and, much like open source code packages, they have driven a new wave of innovation by making information and services just a request away from developers. So much information and so many services actually, that there’s talk of build your app using no stack at all.
And since we were challenging the old stack ways, we also shifted away from the old pull paradigm that the web was built upon. Whenever a client (usually a browser) needed some information from the server, it would be his job to ask for the data – and whenever you hit refresh, it would pull a fresh copy of the data from the server. And that worked just fine back then. But now we’re seeing companies wise up to the advantages of being able to work with fast-updating data, from real time order processing or instant analytics to showing search trends and empowering chat and social interaction. We’re slowly starting to change the way data flows in the Internet, from making simple pull requests to information being pushed in real time to subscribers. Browsers now support this move, and there are plenty of excellent code packages as well as APIs out there that are helping this become a commodity of the future.
What does this all mean for modern infrastructure?
There’s no one stack anymore, these are build-your-own-stack times. And that’s great, because we’re not changing our objectives to fit the platform anymore – we’re free to choose the platform that best fits our objectives.
We started thinking about this after one LAMP-based data management system we built a while ago just couldn’t fit in a world where real time was becoming a requirement. It was designed for pull instead of push, and that’s where the classic data architecture was failing us. So we were hunting for a modern infrastructure setup, that could cut development time for the browser and mobile applications we build today, and we had a few goals in mind:
- It should feel like an API. Having CRUD operations on key-value objects that we can group and filter is routine for web projects. Little to no schema should be required.
- But, as a twist, you should be able to subscribe to data objects, instead of just getting a snapshot of them. We need something that sits on top of publish-subscribe systems, which handle the push of data really well, and transforms that data into actual, synchronized objects that we can work with immediately – without needing to still wire up a lot of boilerplate.
- Something that updates in-memory objects on many devices in real time would be great basically.
- It should be flexible about its approach to persistence, so that we can pick the best database for the job. It should be flexible about transport as well, since we might want to use anything from websockets to cloud APIs or mobile push notifications in our projects.
- Actually, it should just be a smart glue layer in-between existing components, and try to own as little functionality as possible. No new package manager needed.
- Authentication, user management and handling permissions also amount to a lot of boilerplate usually, so we’ll add that to the list.
- Oh, and it should scale. Data changes fast, especially if you have a successful app, so this system should be able to handle data updates from a large number of clients, while keeping objects in sync on every device and not dropping any information on the way.
Essentially, we were looking to upgrade the API to the real time age, where it needs to manage not only data, persistence and security but also collaboration and real time transport of data, amongst others. To do that in a scalable way, you need to rethink the architecture and separate some things, but there are a lot of great components and services out there that fill in most of the gaps, so all that’s required is an orchestrating layer in between, that handles object synchronization. So we built Telepat.
Why this matters
The underlying quest for Telepat is to create the kind of software that can, like LAMP once did, stand at the core of a new generation of apps, working with instant data. Yes, we need the feedback and support of the community along the way, so feel free to get in touch, or join in the discussion in the comments. We’ll be posting updates as we make this come true.