Pure Storage: Flash-based storage powering Meta’s new AI supercomputer

Facebook parent company Meta Platforms Inc., recently announced that its research team built a new artificial intelligence (AI) supercomputer known as the AI Research SuperCluster, or RSC. Meta claimed RSC will soon be the world’s fastest supercomputer. What is perhaps less visible, but stupendously valuable, is the storage partner — Pure Storage — that has been helping Meta achieve its AI research goals.

An IT pioneer that delivers storage as a service in a multi-cloud world, Pure Storage is a long-time technology provider for Meta. The flash storage provided helped to design the first generation of Meta’s AI research infrastructure back in 2017. Since then, Meta has continued to partner with Pure Storage, and the RSC is their latest collaboration.

Meta emphasized that it chose Pure as it needed a storage partner that can deliver robust and scalable storage capabilities to power the RSC. After all, the technologies powering the metaverse will require powerful computing solutions capable of instantly analyzing ever-increasing amounts of data. 

In conjunction with the recent collaboration, TechHQ got into a conversation with Pure Storage’s Field CTO for EMEA, Patrick Smith. 

TechHQ: Tell me more about Pure Storage and what do you guys do?

Patrick Smith (PS): Being a 12-year-old company, we’ve grown quickly and from day one, we’ve all always been focused on making use of flash to deliver new ways of managing data. So today, Pure has a reputation as a pioneer that introduced flash storage into the world of modern data. The company was able to redefine how its customers use and interact with storage, and how it allows them to do more with their data. This is especially essential given the fact that these days, we live in a world driven by data — so being able to interact and consume that data to support business services is absolutely essential. 

THQ: What is the perspective of Pure Storage on the impact RSC will have on the metaverse?

PS: It is a big announcement that resonates with the whole technology community since everyone has an eye on what Meta is doing. What is interesting for us is that we have been working with Meta for some time now, since 2017 to be precise, we have been partnering with the AI research team at Meta US where they have been utilizing our technologies to support their services. The recent collaboration is basically an evolution of that. We were indeed very proud of the fact that a company like Meta, who are themselves innovators, is making use of Pure Storage platforms to underpin the RSC. It shows that they see in our product, something that allows them to differentiate, since this is an incredibly competitive space.

In terms of the metaverse in general, I think it is very interesting because although it is a new trend, everyone is talking about it. There are examples of the metaverse that are already out there. One of our customers is Roblox, the children’s gaming platform that can be viewed as an example of the metaverse. That means we are used to being able to support such environments. 

THQ: So do you foresee the metaverse being more commercialized in the near future?

PS: I think in some ways, like the Roblox example, it already exists. So I think it’s going to be revolutionary. It is going to obviously increase in focus and with time, more consumers around the world would interact with it.

THQ: Can you share more about the technology components behind it that Pure Storage is reliable for?

PS: The fact that we have always been based on flash technology instead of spinning disk, gives us a big advantage. We have built technologies that allow us to take reliability to the next level, like our management platform known as Pure is actually a SaaS-based offering. It collects telemetry data from arrays in the field which allows us to understand when issues might occur, so that we can fix them before they cause a disruption to business services. 

In that sense, it helps us increase the reliability of our services even more. Another thing is in today’s interconnected world of data, operations are ongoing 24/7 and there’s no more time for downtime. Hence, reliability isn’t just when things go wrong and one of the things we are focused on with our platforms is to make sure that management is always non-disruptive. It never interrupts the business service, whether you’re updating the software, whether you’re growing the capacity, or increasing performance, we can always do it simultaneously. 

THQ: Briefly, how does the storage as a service (STaaS) model work? What kind of customers will benefit the most from it?

PS: It is really interesting. The “as-a-service” is pretty general across technology sectors, and storage service to us, is incredibly important for our customers. In terms of who benefits, it is all the way from commercial customers to large global financial organizations. I think it’s important to realize that it’s a commercial offering, and also an operational model that simplifies how customers consume and interact with their storage. 

This is because customers no longer pay for unused capacity in their data center. They only pay for the capacity that they’re using to store their data, subject to their reserved commitments. It really is very much a cloud-like consumption model with the reserved commitments, and then the ability to flex up and down beyond that. Additionally, our customers do not have to worry about capacity management as Pure looks after it, making sure that there’s enough headroom in the customers’ environment to be able to support changing business demands. 

In exchange, customers can free up a lot of their resources to do other value-added work rather than managing the storage environment. Overall, we also look after all of the lifecycle management, software upgrades, adding of capacity, among others. So really “Pure-as-a-service” is often viewed as storage-as-a-service, a commercial model. We are actually a lot more than that.

THQ: In the future of reliable, scalable, and high-performance storage, where do you think this market is going in the next 5-10 years?

PS: What we are seeing with Meta shows just how this world is changing. So it’s always, I think, challenging to look that far out. One of the things that is absolutely clear to us, is that the pace of innovation just isn’t slowing at all, it continues to accelerate. We’ve seen that recently with new technology that we’ve introduced in the world of flash that dramatically changes the capacity, sustainability, and environmental footprint of flash technology. 

It allows people to store more data per watt of energy consumed, which is increasingly important for organizations around the globe. In terms of the whole environmental aspect, data sets are only growing as people look to consume and store more and more data. So being able to keep ahead of that curve is incredibly important for us. 

We’ll continue to push flash technology over the next five to 10 years and in that period, we expect to see the tipping point between flash technology and hard disk-based technology, where there is no reason to run spinning disks in that period. We expect to also be able to support not just higher storage capacities, but also in conjunction with higher performance.

Recently launched a new platform FlashArray XL, that is our next storage platform that brings both higher performance and higher capacity to customers’ data centers. So a huge amount is going on there. Another thing that is hard to ignore is the world of Kubernetes, which I think really is a platform that is here to stay. If we think back to the rise of VMware and the virtualized data center, it really feels like Kubernetes is the next step on that journey.

We will become, over the next five years, a de facto part of the data center landscape that every organization is making use of today. You know, we see a lot of organizations making use of Kubernetes. But I think that will grow massively over the next five years.

THQ: With cloud providers offering new, more efficient data-centric services each year, do you consider it as a threat, or growth opportunity?

PS: One of the really interesting things about the cloud is how customer’s use of the cloud is ever-evolving, especially over the last several years. Now I think it’s settling down, we’re seeing organizations that were moving everything to the cloud, now rebalancing that strategy. They’re looking at what are the right services to use in the cloud? 

What are the right applications to move to the cloud, but also taking into account reliability, performance, data sovereignty, and looking at what can live in the cloud, but also what needs to be in their own data center, where they potentially have more control. And so we’re seeing a maturing of usage between cloud and noncloud. 

The other thing we see is that organizations specially organized and larger enterprises, look at the cloud to be able to leapfrog on innovation. They then look at potentially moving those capabilities out of the cloud, again, when they can deliver those capabilities in their own data centers. That’s especially true for some of the modern analytics platforms whereby as soon as they get to scale, they want to move those into their environments to have greater control. So, I think the cloud is evolving. 

THQ: Finally, what differentiates Pure Storage from your other competitors in the market?

PS: It is interesting because before I joined Pure, I was a customer of theirs. So I’ve experienced both sides of the table. One of the things that I truly believe is that we have a current customer-first approach in our culture, and everybody talks about that, but Pure truly lives and breeds that approach. Another thing that I think is incredibly important is we talk about our customers being innovators, looking to do more with data.

For us, it is important to be able to support them in that. We have to have a history of being innovators and pushing technology boundaries, and it’s just the way we operate. It’s highlighted by the fact that we’ve been leaders in the Gartner Magic Quadrant for the last eight years, and that is quite an achievement for a company that’s only existed for 12 years. Using the magic quadrant is, I think, a real testament to help differentiate ourselves.