Infinite storage: we are almost there…

Seven years ago, I wrote a blog post called What is infinite storage? The blog post was a response to a Physics professor and colleague of mine who objected to my claim that we are soon reaching “infinite storage”.

To be precise, my claim is that as far as computers are concerned, we will soon be able to assume that there is no limit on what we can store. Formally speaking, this can never be true. But, on the face of it, nothing is ever infinite. What scientists mean by infinite is “so large that it might as well be infinite”. For example, as far as my family in concerned, there is an infinite supply of bread and milk at the local convenience store. In theory, we could drink so much milk that the convenience store could not supply us… but it can’t realistically happen. In any case, there are 3 convenience stores and a supermarket within walking distance of my house. Also, I am rich enough that I can afford to pay for all the milk and bread that my family can consume. So, at some point in the XXth century, we reached the “infinite bread and milk” threshold.

So what about storage? I have defined “infinite storage” as “the ability to buy a cheap 10TB disk at my local electronics store”. Why 10TB? Because that’s how much storage you need to record everything you see in a year. You know these Google glasses? They won’t be able to record everything you see in a year… but they could if you could find a cheap 10TB memory chip. They could also record everything you see in a week if you had cheap 256GB memory cards.

So where are we? I think that, in the last two years, we have made substantial progress. Seven years ago, I wrote:

Currently a portable 1TB drive can be had for $569.

What about today?

We still don’t have cheap and portable 10TB drives. They are apparently coming in 2014 according to some reports. Still, you can record everything you see in a year for $600 using two 5TB portable drives. That’s pretty good!

Effectively, this means that you can install a camera in your house, and keep a record everything that ever happens… for a mere $600 a year.

However, these portable drives are a bit large. You can’t imagine hooking them up permanently to a Google glass device. The SD cards are closer to what I want, but to record everything you see in a year on memory cards, you would need to spend 20K$. Basically you would need to buy a new SD card each week at $500 a piece. Some of us can afford it, but I cannot.

What would happen if many of us started wearing glasses that record everything? I bet that the prices would fall and we would soon all be able to afford it.

Why would you want to record everything? That’s a separate topic, but I can imagine that people who are losing their memory might greatly benefit from such records, especially if you have the computer power to process the data and replay the significant bits at will. Maybe this is what Google will be offering in 10 years.

As prices fall further, it will soon become cost-effective to record everything that happens around your car while you are driving it. I think it is only a matter of time before insurance companies entice us to record everything in this manner.

As storage becomes infinite, software will continue to become more important. I would bet that the winners will be companies able to recruit the best minds of the software industry. We will soon be limited by how fast software engineers can solve data problems.

If these ideas sound crazy, consider that Google is keeping a record of all emails I have sent and received since 2004. All 15GB of it. Today, storing 15GB of data in the cloud is trivial. Ten years ago, it would have sounded futuristic.

Non-disclosure: I do not own any Google stock. In fact, I do not own any technology stock. I probably should.

10 thoughts on “Infinite storage: we are almost there…”

  1. Clearly, you could take advantage of good old memory hierarchies and dump you SD card weekly to your 5TB drive. You can probably afford a $500 SD card once.

    The recording of everything around the car already routinely happens in high-risk places where the cost/benefit ratio favors it, such as around some police cars and in Russia.

  2. Your definition of “infinite” seems to be strangely bound by the amount of data collected in *one* year. That appears to be quite shortsighted.

  3. @Daniel

    I think the amount of data we want to store grows faster than the storage capacity. Today you can store everything you see, hear, and read (according to your calculations). Tomorrow you will want to store historical analyses of your data (“how often do I check email?”). By next week you will to store comparative analyses of your data and your spouse’s data. By next month, you will need to store historical analyses of your comparative analyses. And so on…

  4. I suspect that we will start demanding higher resolution video as storage gets cheaper (although this will be bounded by the human eye’s ability to resolve pixels). We’ll probably go up to at least 4K or 6K resolution which would require about 20 storage of what Google glass uses. Also, I’d expect demand higher quality audio as well.

    Of course, assuming exponentially cheaper storage, this would only add another 5 years or so to the estimated arrival of ‘infinite’ storage.

  5. No Mihai, I think Daniel is right on. We have better sound quality than the ear can get since a long time. Moreover, although I remember how hardrive and bandwith were a continuous problem for pictures, scans, music and movies (even simple software at the beginning). The amount I am spending in storage is getting ridiculessly low, enough so that I don’t by that much storage anymore (especially compare to milk 🙂 ).

  6. Gordon Bell’s MyLifeBits project (http://research.microsoft.com/en-us/projects/mylifebits/) aims to do exactly this–digitize the sensory arena of an entire human life.

    From reading some of the papers, the impression I got was that the research challenge shifted very quickly from “how are we going to store all this data” to the retrieval problems in accessing, finding and presenting all that data in a coherently and quickly.

Leave a Reply

Your email address will not be published. Required fields are marked *