Conquering the 44 zettabyte data mountain

The amount of data that will be swirling around planet Earth is set to explode during the final years of this decade and will reach the biblical proportion of around 44zettabytes (ZB). To put that in perspective 1zettabyte is the data equivalent of 250 billion DVDs.

For those not familiar with that term, a zettabyte is a multiple of the basic byte unit and each zetta stands for multiplication of 1021 or, in other words, 1000000000000000000000 bytes.

So 44ZBs is a whole lot of data and of that amount the connectivity giant more commonly known as the Internet of Things will contribute about 12%, and that will come from the 30 billion connected devices that are predicted will be pumping out massive amounts of data by the year 2020.

All this is terrific progress of course but it's not much use just unleashing ever-increasing mountains of data if it cannot be fully used, exploited and analysed. To make data useful it must be stored and analysed very quickly and this creates serious challenges for service providers and system builders who must balance cost, power and performance issues when designing memory and storage solutions.

Enter a new form of memory that promises non-volatile speeds that are 1000 times faster than conventional technology and say the creators, Micron and Intel, is the first entirely new class of memory for 25 years. What it provides is the memory performance that computers and data centres, medical research, computer gaming, space research and a whole lot more applications will have to have as we enter the next decade.

Called 3D XPoint technology, its creators developed special material compounds and a cross point architecture that is 10 times denser than conventional memory and is also much more durable.

One of its key operational attributes is it can delve into data that has been held in long term storage and can very quickly retrieve it, This has been a real problem in the past.

For example, retailers may use 3D XPoint technology to rapidly identify fraud detection patterns in financial transactions and medical researchers could process and analyse larger amounts of data in real time which would accelerate genetic analysis and disease tracking.

After a decade of research and development, 3D XPoint technology will undoubtedly usher in a new class of non-volatile memory that reduces latencies. This is important because it allows much large amounts of data to be stored close to the processor and accessed at speeds previously impossible for non-volatile storage.

At the heart of this development is a transistor-less cross point architecture. Fundamentally this is made up of a 3D structure constructed from layers of wires. On each layer, the wires run parallel to each other and at right angles to those on adjacent layers.

Between each layer are vertical sub-microscopic columns, which connect the points at which the wires cross and memory cells are located at the junction of word lines and bit lines. This means data can be written and read in small sizes, leading to faster and more efficient read/write processes.

There is no doubt that this new memory architecture will be an essential element in how masses of connectivity data is used.

But flash storage and the humble RAM will still have a role to play and it looks very much like 3D XPoint technology will work in tandem with those technologies to ensure that processors can access data far more efficiently as the zettabyte mountain continues to grow.