Proving the Performance of Solidigm SSDs at StorageReview

Stephen Foskett: Analysts and press spend a lot of time talking about specs and performance numbers. So it's always a treat when we get to talk to people who are testing and using these products in the real world. And when I say in the real world, I literally mean out there in the world, as you're going to hear this time. This episode of Utilizing Tech is focused on AI data infrastructure and features Jordan Ranous from StorageReview and is co-hosted by myself and Ace Stryker from our sponsor, Solidigm.

Welcome to Utilizing Tech, the podcast about emerging technology from Tech Field Day, part of the Futurum Group. This season is presented by Solidigm and focuses on AI data infrastructure. I'm your host, Stephen Foskett, organizer of Tech Field Day and joining me today as my co-host is Ace Stryker of Solidigm. Ace, welcome back.

Ace Stryker: Hey, Stephen. Thank you so much. I'm very excited to be here and looking forward to this season of the podcast with you.

Stephen: Absolutely. It's going to be a lot of fun. We're going to be able to bring in a whole bunch of folks all season long from different customers, different practical applications. That's what we're talking about today, Ace. We hear a lot about speeds and feeds. We see a lot of numbers. We see a lot of bragging from vendors, but really nothing matters until this stuff is out there in the field and being used.

Ace: Yeah, I'm really excited about the guests we've got lined up today. In a lot of these AI conversations, what we hear is the industry talking to itself, right? We hear various solution providers and hardware and software vendors hawking their wares and extolling the virtues of their products. Where it gets really exciting for me is to be able to hear from folks who are out in the field, who are using these things in the real world where the rubber meets the road and producing really cool results. So, in that spirit, I'm excited about the conversation today.

Stephen: Yeah, totally. And although they're not, I guess, strictly speaking, an end user, one of the sites that I love, one of the companies that has this incredible video content that I enjoy is StorageReview. I love running into you guys at the show. Jordan, you guys are always playing with the coolest toys. Welcome to the show.

Jordan Ranous: Yeah, thanks for having me on. I’m Jordan, I'm from StorageReview. More officially, my title is the Advanced Workload Specialist. So anything kind of advanced, whether it's AI or HPC, it's my job to take all the cool toys that we have in our lab, put them all together and make them do fun stuff.

Stephen: And that's really what we're talking about here. So this season of Utilizing Tech is focused on AI data infrastructure. You're the guy that's out there taking these things, trying them out, trying to see what these systems will do in terms of AI data infrastructure. So I guess give us a little bit of background in terms of what level of coolness have you put together with Solidigm SSDs.

Jordan: Yeah, so Solidigm has been a longtime friend of StorageReview. We got together with them not too long ago and had a fun idea for some, as Ace alluded to, where a rubber meets the road, almost literally in this scenario, plan to take some of these high capacity, pretty quick SSDs and stick them out in the field and do some real field AI work with them. We came up with a pretty neat concept around astrophotography, which is one of my personal favorite hobbies, and we filled up a Dell XR7620 with four of the Solidigm QLC SSDs and took it out to the frozen wilderness and shot some pretty incredible space pictures with it and were able to take that data and bring it back to our lab and do some AI work with them.

[Read about that project here: Solidigm™ High-Density SSDs Enable Edge AI in Boon to Astrophotography]

Ace: So Jordan, what's the connection there? Where does AI play into this? I've been out there and I've done a little stargazing. I've got a little telescope that the kids and I look through at home. Obviously the images you're producing are much higher quality. Is it the AI that enables that level of quality or how are you using AI models to kind of develop your output?

Jordan: Yep. So there's a couple different ways that we were taking advantage of it. The first aspect, we were using the high capacity SSDs to capture all of the data. Our images are about 62 megapixels each raw, and that's before you add color and chroma data to them. From there, we were able to use that and find tune manually; go through the data and comb through it and get a really good subset of images. [We] combine that with some actual Hubble legacy data as well and run them through a novel convolutional neural network (CNN) to create a more advanced, modern, CNN-based denoise and sharpening algorithm that outperforms the traditional kind of Richardson-Lucy algorithms that you see out there. Because of the amount of data and the amount of time that we were able to spend out in the field, it helped us really kind of drive that model forward, which we can then bring back out into the field and do edge inferencing on the images real time to see if we're having some other issue. Maybe we're slightly out of focus, or there's a little too much dew on the lens, or there's a vibration happening from a car driving by. And we need to know about that so we can make an adjustment if there's something, [like] stray lights. This is a very hyper-specific task where we focused in and we were able to take that time to [get] actionable results and shrink that down by using a neural network in order to help make decisions in real time out in the field.

To add a little bit of additional color to that, traditionally, you would take a photo and you could spend hours processing through it and working through the sharpening or the processing and the stacking, only to find out that the entire weekend that you went out on your camping trip and took all of your photos, it just didn't work out for some reason because you were slightly out of focus. We're helping to drive methods that can bring, like I said, that actionable real-time feedback into the scenario. And this has a lot of applications beyond just the astrophotography side.

Any sort of imaging is actually compatible with the network that we put together. It was just specifically that we trained it on the space photos because that was something that was near and dear to my heart and something that I'm fairly decent at or at least I like to think to in order to prove out the idea and flesh it out.

Stephen: It sounds to me like you've basically created sort of an edge computing workload or test case there because you've got poor connectivity, bad environmentals maybe, and yet some high throughput AI applications that are running there. So if you sub out telescope and stars, you could easily think that this similar situation could apply in energy, medical, military, all sorts of different areas, and yet you might not be able to talk about those use cases because they're sensitive. Whereas what you're doing is something that you can try out and experiment with and play with in a way that is wide open, that you really can have that conversation, right?

Jordan: Right, the stars aren't going anywhere, and if we mess up the AI, it's not going to turn off Andromeda on us. But yeah, a couple of the big things that come to mind, like you said, the other industries, self-driving cars, providing more real-time image clarification to the cameras that are needed on a self-driving car, for instance, or exploration. Oceanic exploration is another one where you're capturing a lot of data and you may need to move quickly and having the ability to go back and clean it up in a meaningful way is really important because of the value and the amount of costs associated with that data acquisition.

Ace: Jordan, a lot of what we're exploring in this season of the podcast is around data infrastructure and what are the kind of requirements needed to support use cases like the one you're talking about. So we talked a little bit about what's happening at the edge and the inferencing that's going on there. Can you speak a little bit to the upfront work of developing the model and sort of what your infrastructure requirements look like there, maybe particularly around the storage and take us through the model that you ended up using?

Jordan: Yeah. So without getting too nitty gritty into the math of it, there was a lot of backend work that happened in our lab initially. We were doing some training on some Supermicro blade systems. We were doing some distributed training, and that involved putting together some really fast SSDs to be able to work in the data in and out of the GPUs as fast as possible. Our data set wasn't exceptionally large, but we ended up requiring large amounts of VRAM due to the nature of the image processing that was actually happening. We were, I think, at over 2000 layers for the neural network at one point, which was able to eat up enough VRAM to span four H100s.

So getting that data in and out of the GPUs, specifically the model checkpointing, and being able to take all of those different checkpoints and save them out so we can go back after the fact and assess the performance, was one of the keys that we needed. We had selected some of the faster SSDs that were maybe a little smaller on the capacity side because we were only working with 320 gigabytes worth of VRAM that we had to then condense out and flush out for the checkpoints. But the key there was the speed in getting the GPUs because we only had them for a short amount of time and getting the data back out of the GPUs and getting those checkpoints saved.

Ace: So, what's next for you in the world of astrophotography? Is this a project that continues to live on here? I've seen the images on the StorageReview site, they're gorgeous and I highly encourage our listeners to go check those out. But I'm curious about, are there bigger challenges you're looking to tackle in the astrophotography realm specifically, as well as, are there ways for others with similar interests to leverage some of the work that you've done recently in the space?

Jordan: So, the next goal immediately on the horizon is to finish up the paper with my partner in crime on this model and get that pushed across the line. And then actually open source this model out for folks to be able to use in some of the existing software that's out there right now and really put it in the hands of the developers who can help make those implementations in some of the open source capture software happen and help give the real time feedback to the users. The whole idea here is to help improve the hobby as a whole and not get too hyper focused on one specific thing, but just help everybody out with the network. Because at the end of the day, it is actually relatively small to run and being able to take all this information and all this time and data that we spent capturing it and condense it into a really small, really efficient model, I think that's one of the more rewarding aspects of doing this project.

Stephen: So what did you find in terms of, what were some of the nuts and bolts, nitty gritty, kind of the fun things that you found when you were experimenting with this setup?

Jordan: Yeah, so I'll touch back on my edge server that we used for this. Dell sent over a XR7620, which is one of their ruggedized platforms, and we had it in a big ruggedized road case that was impact resistant, weatherproof, and shock resistant. And when we were looking at kitting it out and kind of building everything up, obviously a GPU was on the list, we ended up with the NVIDIA L4 in there, they had a [Ada] Lovelace 70 watt card or 75 watt card, I believe, but it's 24 gigabytes of VRAM to be able to work with. So we were able to do some kind of real time work at the edge. But as far as fine tuning goes, when we started looking at storage solution, what's generally provided we found to not be quick enough for what we were trying to do. Traditionally, when you're going out to capture this stuff, you're going to be a guy with a laptop and a camera and a USB cable connected to your scope. You don't need all this overkill stuff.

But when we start getting into, “We need to collect a lot of data. We need to look at it in real time. We need to be able to make decisions on are we doing what we're doing properly? So we are capturing the best data to provide the best training data.” All those factors combined led us to needing a lot more higher throughput from both CPU perspective as well as our storage perspective. So we did go with four of the U.2 NVMe SSDs. We had two flavors of the [Solidigm D5] P5336’s. We had the 60 TB guys and then we had the 7.68 TB drives in there. Both of which performed great.

The main reason why we ended up going with these, especially for the high density, was actually the old sneakernet story. Getting the data back to the data center for larger processing later, it was more efficient for me to fill up one of these 60 TB drives and drop it in a FedEx mailer and have it two day [shipped] back to the lab where I could then interact with it on big irons, so to speak, as far as the H100s and the bigger AI machines go, than to try and upload it over limited bandwidth, because we actually ended up doing this out in the middle of nowhere on the banks of the Great Lakes.

Ace: You're someone who, from my point of view, very much has their finger on the pulse of what's going on in the AI world by virtue of your work and your personal pursuits as well. I'm curious about this general trend of more and more AI work moving closer to the data, moving closer to the edge. You mentioned in your case there's a benefit of having a better sense of how you're doing while you're out in the field and what your outputs are going to look like so you don't find out down the road that you've ended up with nothing usable. Can you talk about maybe other ways you're seeing that trend manifest across the AI landscape today in the course of your work and your interactions with lots of folks across the industry? How prominent is this trend and how quickly are you seeing AI work moving from the core data center where a lot of it has lived historically to where more intensive work is happening at the edge?

Jordan: Your question almost is the holy grail and what everybody's searching for right now. The feedback that I'm hearing going to the shows and interacting with the viewers and the people who ask questions about these products coming to me for answers are, I've got my data center full of GPUs and I think I have an AI model and now I needed to deploy it to 1500 retail locations. Or actually, just the other day, this was kind of a fun one. I went through a McDonald's drive-through that had an AI ordering for the first time. That was kind of a wild experience actually. But the name of the game for everybody is we've got all this data. We've got it right now very centralized and a lot of folks are taking the similar steps that we did with this project, which was, “Okay, let's keep that data out at the edge. Let's inference on it out there and then feedback just the key metrics into either the data center or into the management interfaces, real-time dashboards, that sort of stuff, and just getting it down to that real granular level really quick.” 

Traditionally, with the big data stuff working in large data bases, a lot of those things have to deal with ETL [Extract, Transform, Load] jobs, they take a long time, whereas if you can process that stuff and save that important data out at the edge, even though you might not be able to send it all the way back to the data center right away at the time of capture, having that real-time performance at the edge in order to be able to make a decision on it as a business, is where things are going and I think it's what a lot of people already see is this is what we need to do with AI and this is how AI can actually help the business. We don't have to move truckloads of data. We can move metadata or a few values or a few model outputs for analysis and then worry about moving the heavy stuff later.

Stephen: Yeah, that's something I really want to zoom in on there, Jordan, because you're completely in line with what we heard when we did previous seasons of Utilizing Tech. We focused on AI. We talked about industrial IoT and industrial vision applications. We talked about media production and all of these industrial IoT applications. That's exactly what they're looking for. They're looking for a way to move processing to the edge. Not just data collection, but move processing to the edge. To use AI as a way to collect, not just to process data, but to collect more data, to process more data, and to get more value at the edge and then ship back the anomalies or the interesting aspects, the interesting elements. We're seeing that more and more, I think, in a lot of these edge use cases. Think about pretty much any kind of IoT vision environment. They're collecting all sorts of camera and sensor data all the time. The last thing they want to do is just be writing all that data and then taking it back home. Instead, what they want to do is they want to have intelligent processing of that data and then take the good bits back home. And similarly, a lot of the things that you're describing to me, it sounds like a lot of the same constraints that people are facing at the edge. You're talking about ruggedized servers, you're talking about power. I think that's another aspect here too. If you're going to have a fairly high-powered GPU, you need to think about cooling. You need to think about power requirements. You need to think about adverse conditions that these things may be installed in. [The] back of a pickup truck [is] pretty adverse, but actually not that wild when it comes to things like energy exploration or military applications or things like that. That's actually a pretty nice environment compared to what the military faces. And also, if you've got something like SSDs that have large capacity, large performance, and a low power footprint, that really helps. I know that your stuff, for example, is battery-powered, right? And so having the ability to reduce that power envelope gives you hours more processing time, right?

Jordan: Yeah, I mean, the difference of 25 or 50 watts that you would have to spread out across spinning rust versus consolidating it into a single large capacity SSD at the edge, certainly that can mean the difference of hours depending on your system, especially if you're loading up the GPU. You brought up something quite interesting there, though. We did a demo with Solidigm actually, back at FMS in 2023, that we did just that where we had a camera up and we were capturing all 1080p 60 fps of the camera and saving it down to large capacity hard drives that could then be yoinked out, shipped back to your data center, but we were doing real time inferencing on it. It's a lot cheaper in more than one way, not just financially, to send a single line of text back 60 times a second than a whole video frame or a whole 4K frame even. So when we look at that, that was kind of our proof of concept of this idea that everyone's been talking about, that everyone's been saying, this is what AI can do for you. This is how you can use it. You can do that edge inferencing and take back the data, but you don't have to throw away. That's where the larger storage stuff comes in. You don't have to throw away that raw data because that's valuable.

At the end of the day, everybody's data, they want to save every bit and bite of it because who knows what the next evolution of AI is going to look like. Oh, you can just point it at a drive full of stuff and it'll figure it all out for you, maybe. But without saving that, if you were just throwing it away because you didn't have enough storage or your storage wasn't fast enough to keep up, right? That's where you kind of get the double-edged benefit of things like these huge QLC SSDs at the edge. And you mentioned the rugged thing, I've got a little fun anecdote. There's a video floating around on the internet somewhere of me actually running these things in a blizzard. And I had sent it to my PR friend at Solidigm and I think the first question was, “What's the temperature?” And the response was, “I'm pretty sure we're not rated for that.” But they were fine. They were rugged enough, they could handle it. That's not something where things with moving parts would necessarily be able to handle it. But we had to go through it as part of our capture when a storm ran through and the rugged hardware chugged right through it.

Ace: Jordan, I want to pivot a little bit here. Because one question that a lot of folks are asking; we heard it at GTC earlier this year, we’ve heard it in various forums and conversations with our customers, is how do you evaluate storage performance for AI? I know this is an area where you've done a lot of work, so I'd love to pick your brain here a little bit. Where I go as a PC guy is if I want to measure CPU performance, I run Cinebench. I can make an apples-to-apples comparison between a couple of processors, or I can run PCMark on a system level. There's a lot of interest in understanding how that's done for storage for AI in the data center specifically. What's your view of the state of things in that space now? Are there emerging tools that are designed to serve that purpose? How well do they work? What are you learning there, and what do you see as recommendations for folks with an interest in understanding that a little better?

Jordan: I thought we talked about no loaded questions! There's a lot to unfold there. If we think through our phases of AI training, you've got data ingest, data prep, the actual training that goes into it, checkpointing, and then out to inferencing once you're done with your model. When we look at those five different phases, there's different needs for each of those phases. When you take that into consideration, you don't want to have to go buy out five different SKUs of an SSD to fill up your data center, but you need to thoughtfully design, all the way through, the platform that you need for your business. Every AI is not going to be the next Llama or ChatGPT or Dolly. Everything is going to be unique. What we do right now in our lab is we're taking a look at a lot of different types of storage; QLC, TLC, throwing in cache layers, utilizing the CPU and the DRAM. [We’re] going GPU direct and kind of profiling these different workloads that we're seeing coming out, whether it's synthetic benchmarks or open source projects. [We’re] looking at how those are actually impacting the system, how they're treating the desk, how they're working with the system memory with the CPU. And then [we’re] mimicking that using some open source tools. GDSIO [GPUDirect Storage benchmarking utility tool], for example, is really powerful. If you know what kind of AI you're going to use, because there's 15,000… I'm sorry, 16,000… no, 17,000… no, 18,000 now… kinds of AI, because it's always changing.

But if you have a general idea of what you're going to be doing, you can actually go out and profile and set up these tests. That's what we're aiming to do with a script that we've been working on. [We want] to look at storage from a total perspective and Gantt chart it out and say, okay, this type of device or this specific NVMe drive is really good if you're doing this type of AI. But if you're doing this kind, you need to look at a storage appliance like this. Or if you're doing another kind of AI, shoving your GPU server full of as much NVMe as you can get in there is the way to go. And then let the CPU worry about offloading those checkpoints over your network at a later date. When we see stuff like, Nvidia just came out with the, and I've got a video on this there, 800 gigabit networking. We're starting to see that be less and less of a bottleneck. But then the storage servers and the storage appliances are going to have to start to catch up to that too. 

We were doing a test recently. I can saturate 200 gigabit Infiniband, very easily with a single A100 GPU. And then we start talking about H100s and now the Blackwell stuff's on its way. You're going to need those bigger, faster, stronger [SSDs]. And that's where [PCIe] Gen 5 gets really exciting, as well as Gen 6. I know Gen 5, right, when we look at the speed on there, the name of the game at the end of the day is to keep the GPU working as fast as possible, as well as non-stop as possible. Everybody knows that. Selecting your storage infrastructure and what you're going to be doing around that is something that's getting more and more focused pretty much every day right now. We need a way to, if we're doing really read-heavy stuff because we're doing ETLs, but we're using the NEMO framework that does the real-time augmentation of the data, so we're not over-fitting our model or something like that. We need a lot of random 4K read performance. And so there's layers to it, right?

And that's kind of what we're aiming to do, at StorageReview at least, is provide the, maybe not the textbook, but give out the playbook for folks to be able to profile their storage or make a smart decision on either their existing infrastructure or some infrastructure that they might be looking at and say, this is where this fits, and this is why it's important for that. You need the density above all for this reason, or you need the Gen 5 speed above all for this reason. And providing that is, I think that's kind of what everybody's looking for right now. So, yeah, it's a loaded question. I think we're on to it as far as getting these GSIO stuff up and going, getting our own FIO scripts, because you can mimic a lot of these workloads in FIO. There's multiple ways to skin this and to make it happen. But I think we're on the right track.

Stephen: It's interesting you mentioned this, Jordan, because I've actually found that it is actually, this equipment is getting so fast and so good that it's actually kind of difficult to push it. How do you test a 60 TB SSD that can push gigabytes per second of throughput? How do you test a processor that has over 100 cores? How do you test terabytes of memory? You don't find that that's a challenge?

Jordan: You're bringing back memories to when the AMD 96 core Genoas came out. That was actually one of my first testing tasks that I had when I joined StorageReview. “Hey, go test these 96 core CPUs. By the way, here's a terabyte and a half of DRAM and a bunch of 30 TB SSDs in it.” I like the approach of we look at it as each individual piece of the system and profile that, right? So when we looked at our CPUs, we decided to do something crazy with the CPUs, where we run the traditional benchmarks that could scale, that could handle that level of thread and that level of core. And then [we] pit those tests down both the product stack currently and historically to be able to show that scaling. We started talking about SSDs. We got to look at the total performance of them by absolutely hammering them with writes.

So we took one of our more traditional tests, right? So for these big SSDs and big CPUs, we took Y-Cruncher and then we needed to test SSDs with it. There's a swap partition in there. We just did 105 trillion digits of pi with that, set that world record, and then we just did 202 trillion digits of pi on the 60 TB drives and took that world record. But that was all basically in the name of let's put these things up there. And Solidigm was willing to send them to us and say, our SSDs can survive this level of abuse. We put absolute petabytes through these things when we were doing it. And it was just that total system test. But then I can go out when people approach me at shows and say, hey, is that QLC good enough for my workload? I can say, well, if you're going to put 20 petabytes through it in a year, yeah, you'll be fine. It'll last you maybe 10 years. That's a pretty good thing to be able to say. We did it. Here's the raw data. Apply it for yourself. See how it works out.

[Read about that project here: Another Serving of Pi: Solidigm SSDs Help Calculate New World Record]

Stephen: Yeah, that's so cool. It's a crazy ability that you guys have over there at StorageReview. That's one reason that I'm a reader and that I enjoy looking at what you're doing. Thank you so much for joining us on here today. Ace, before we go, how would you summarize this? Real-world testing, real-world benchmarking, proving out AI data infrastructure with StorageReview.

Ace: Well, it sounds like everything else in AI. It's moving rapidly. And I think what we're learning here is that the possibilities are opening up in terms of what you can do at the edge, which is really exciting in a lot of fields. From medicine to manufacturing to energy. And so that's a trend that is going to be driving a lot of innovations and probably touching our daily lives in more and more ways going forward. And so I'm thrilled to hear some of the developments there and firsthand from you, Jordan, about your own findings and some of your projects. Very, very exciting stuff. Glad you also found a way to plug the Pi project in there as well. I think that's super cool. That's become one of my kind of go-to party facts. Hey, did you know the 105 trillionth digit of Pi is a six? And maybe that's why I don't get invited to too many parties. But anyway, I find it super interesting. And I appreciate your thoughts on the benchmarking piece as well. That's something that is not as easy as it sounds. There's so many variables in there. It depends on the use case. It depends on the architecture. But I think it's of interest to a lot of folks in this space, as we look forward, that that continues to mature and that the industry finds a way to measure and communicate storage performance within an AI infrastructure in a way that enables folks to make easy A to B comparisons. And so we'll certainly look forward to your continued work on that front as well.

Jordan: Yeah, it's been great to be able to go through and test all of these different kinds of permutations and the stuff that everybody's talking about when you go to the trade shows and you see the keynotes and you see all the booth demos. It's really fun to see those and then actually to be able to take them out into the real world. I feel like I'm one of the luckiest people in the world. I had the best job ever. I get to take this fun stuff, take it out, actually, like you opened with, make the rubber meet the road, put it to the test. And it's been absolutely great partnering with Solidigm. They're long time friends and great friends to work with in the industry. We can throw some links in the description to our AI story as far as the CNN goes. And we can throw in our pi records as well for you guys to take a look at. [Visit] storagereview.com for all the latest and greatest in data center hardware tech news and reviews.

Stephen: Thanks a lot, Jordan. Ace, before we go, where can we continue this conversation with you apart from listening to Utilizing Tech every Monday?

Ace: We'll be very busy this summer at events all over the place as well, so keep an eye out for us at all the major OEM conferences and industry events going forward. You can continue to track the latest on our products and work that we're doing through partners at solidigm.com/ai.

Stephen: And as for me, you'll be seeing me at Tech Field Day events this month, and after the bit of a summer break here, we're going to be doing an awful lot of stuff. Of course, you can also catch me here at Utilizing Tech every Monday, the Tech Field Day podcast every Tuesday, and of course, our rundown of the week's news every Wednesday at Just All IT. Thank you for listening to this episode of Utilizing AI Data Infrastructure, part of the Utilizing Tech podcast series. You can find this podcast in your favorite application, just look for Utilizing Tech, as well as on YouTube. If you enjoyed this discussion, please do give us a rating or a review. We'd love to hear from you. The podcast is brought to you by Tech Field Day, home of IT experts from across the enterprise, now part of Futurum Group, as well as our friends at Solidigm. For show notes and more episodes, head over to our dedicated website, utilizingtech.com, or find us on X-Twitter and mastidon at Utilizing Tech. Thanks for listening and we will catch you next week.”

Copyright Utilizing Tech. Used with permission.