This is a write-up of a project from my archive. A prototype built over a couple of days in 2019 as an experiment. I didn’t publish it then, having not taken this specific project further than this two-day hack. But I’m now sharing the process, codebase and learnings for others to build on and who knows, maybe I’ll share the more fully-fledged project that this spark spawned soon.
We live in a connected world. With more and more digital services intersecting and blurring the lines between one another, it’s incredible to see how much untapped value you can access by connecting these services just one degree further.
As an example of this, a few years ago the digital bank Monzo announced a partnership with Flux, a company aiming to digitise the paper receipts of the world. As if by magic, a purchase at a participating retailer leads to an in-app receipt, fully itemised and without any need for user involvement. Even more impressive, in trying the service, it was able to resolve prior purchases from before I enabled Flux.
Flux is a very polished and well constructed service, but of course, isn’t able to include every single merchant from the outset. So, in order to fill that gap, we need to look elsewhere. To a single place that accumulates all of the digital receipts you obtain… Your email inbox!
So. This is where we need to start getting creative and where this article becomes more hands-on. I wanted to build an automated service that would look for new receipts in my inbox and associate them with a transaction I’ve made on my Monzo card.
So. First thing’s first. We need to get our emails out of gmail and start pulling them apart!
Let’s speak to gmail
To try and keep this simple, I initially tried using IFTTT to listen for inbound messages (on a specific filter), this didn’t really work as well as I’d hoped as IFTTT doesn’t really get the markup of the email back (or at least can’t pass it along), so this was out of the question.
So, I built a Node JS client that subscribes to the gmail API via Google Cloud PubSub, with two modes:
- Listening for new emails with the search term: “label:^smartlabel_receipt” — which will use the built-in features in gmail to identify mails that have a receipt!
- Searching through your inbox with given parameters for receipts (this is useful for updating older purchases)
Now that we’ve got a set of potential receipt emails, we need to start working out what we’ve actually purchased.
Wait? I bought what?
So, now we’ve got the emails and we’re ready to start understanding them. For this we’re going to need to parse the HTML. I brought in a library called cheerio to give us a robust mechanism for parsing the email markup into a DOM.
Now that we understand the markup in the email, next came the process of pulling it apart! After sifting through a few emails, I started to write some query selectors (expressions that point at specific places in the document) to select the elements containing the rows of items, total price, order ID and other data we need in order to make a Monzo style receipt.
In order to support a wide range of merchants, I’ve tried to make the parsing plug-able, with a really basic plugin system where multiple parsers attempt to parse the email and return a nicely formatted Monzo Receipt if they can.
👋 Hey Monzo
So, now we’ve got a receipt ready to go, we need to pass it over to Monzo!
First step. I made a custom client on the Monzo API (which is in pre-release, so use with caution), this allows us to perform an OAuth flow to negotiate the authentication credentials we need to start adding receipts to purchases.
Now that we’re signed into Monzo, there’s two things we need to make sure we do.
- Find your transaction
Cause at the moment, we have no idea which monzo transaction your receipt belongs to!
- Add a receipt to that transaction
Send Monzo the receipt to attach.
So, I wrote a super small Node.JS client for Monzo that takes our credentials and our receipts and performs the above steps. Resolving the individual transaction first and then following by updating that transaction with the receipt.
And that’s it. We’re done! Our receipts are automatically matched and visible inside our transaction in Monzo 🎉
Where can I get this?!
This project was an experiment, put together over a few days back in 2019. It serves primarily as a technical proof of concept. The codebase for this experiment is available on GitHub here as a point of learning or reference for anyone looking to implement something similar.
Due to the broad scope of permissions necessary to implement this functionality, I opted not to take the project beyond the initial prototype.
Although this proof-of-concept didn’t go beyond that, I did take this core idea forward into a more robust personal project, which I may be sharing in the near future…
The lines are becoming even more blurred between the digital services that we consume. Fuelled by the rapid pace of innovation across all technology consuming sectors, integrations such as this will only become more common.
Though, this bleed of data from one distinct category into another highlights the already fairly worrying privacy concerns. If you weren’t scared enough about how much Google knows about you, how do you feel about your bank knowing the same information? That’s forgetting the service in the middle that has access to both pools of information. For me, this is where the ease of integration becomes essential. Having the ability to “DIY” your own version of these integrations in a relatively short time frame is really exciting and helps to put the power back in your own hands to a degree.
The ability for integrations such as this to bring your digital life together, highlights how easy it’s becoming to piece together technology that is almost invisible. I can only expect that the future will bring the partnership of more cross-industry services that, as if by magic, silently make our lives easier.