Allen's tech crib
3 Oct 2014
A simple auth implementation for a REST based service

I’ve been working on a project in my spare time. I am planning on using this platform as a means to share, in semi real time, the technical challenges that I’m working through, and my reasoning for certain decisions that I make.

While building the backend for a prototype system where users can make requests and workers can fulfill them, I needed some concept of authentication.

The requirements are straightforward and standard:
-Users and workers need to be able to register for an account
-Any request made “belongs”, semantically and in implementation, to a particular user. When I make a request, that request belongs to me.
-ACLs. Only workers can fulfill requests, a user can only change his/her own account details, and so on.

I went with a simple auth token model, as it is common, easy to understand and implement, and secure. Here’s how it works. When a user logs in (or successfully creates an account) through a client, the server provides an authentication token, a string that “proves” that the user is who he says he is. Any request made by the user through the client to the server has to contain the auth token as an HTTP header. For the server, given any incoming request, its auth token header would identify the user making that request. As long as user types are saved in storage in some way, the server would be able to map the user making the request to the type of the request, thus implementing ACLs.

For storage, I am using Redis, a very simple but powerful key value store. I’m storing users as userId:userObjectAsJson. The rationale behind this choice was that when doing a user fetch, the majority of the time I’d like all the user details. The same is true for request objects. I am also storing a reverse key of email:userId. The reasoning is twofold - the login treats a user’s email as the username, so fetching the user object (from the userId) needs to be an O(1) operation, and to check for duplicates (two users cannot share the same email address/login username). 

For authentication, I initially had another reverse key that mapped auth tokens to user IDs, for simplicity’s sake. A qualm I had about this was the fact that the key space would increase by one per every new user, which should be avoided if possible. What I ended up doing for generating the token was taking the user ID, concatenating it with the timestamp of the authentication/registration, symmetric key encrypting it, and base64 encoding it (to minimize the storage footprint). This way, given the symmetric key, auth tokens can decoded to 1, the user, and 2, the time of last authentication, for expirations.

It’s important to note that an auth token is equivalent to a password in terms of potential damage and ability to control an account. Using a https endpoint prevents MIM attacks.

3 Oct 2014
Storing hundreds of millions of simple key-value pairs in Redis

A clever hashing approach to reducing keyset and storage space using Redis.

Using the hashing/bucketing approach, they managed to bring the storage required for a 300 million sized key space from 21GB to 5GB.

Awesome to learn that Instagram uses Redis [for reverse mapping of media IDs back to the individual users]. I’m becoming a big fan of Redis!

31 Aug 2014
Install Redis 2.6 on Amazon EC2 Linux AMI or CentOS

Concise and simple to follow. This walkthrough includes instructions on how to configure and deploy Redis on an EC2 vm.

Redis isn’t a builtin service on EC2, like tomcat is. Amazon probably wants people to use elasticache, their kvstore.

30 Aug 2014
An Android REST Client and Tomcat REST Webservice | Avilyne Technologies

Great tutorial on how to set up a tomcat server. It covers the basics of building a REST application using Jersey, including examples of how to use beans for JSON serialization/deserialization.

Very first timer friendly, unlike some other guides out there that assume some domain knowledge. It’s also a bit ambiguous for a newbie what jars are required, and having to fetch them when you’re not using maven wastes a bit of time if you’re just prototyping; this guide is inclusive of this.

8 Apr 2014    3 notes
WTF is a SuperColumn


Hey there: this post is pretty old is based on Cassandra version 0.6. Some of this still applies but for the most up to date documentation check out
For the last month or two the Digg engineering team has spent quite a bit of time looking into, playing…

With all the different noSQL solutions out there, I was a bit confused by all the different options. While building out a prototype for Proxy, we looked at columnar stores such as Cassandra and BigTable.

This is a good article explaining what exactly this type of data store is.