The National Academy of Television Arts and Science has honored Dell Technologies with a 2020 Technology and Engineering Emmy® for the Dell EMC Isilon for driving early development of Hierarchical Storage Management (HSM) systems. This is exciting news for those of you working with artificial intelligence (AI), machine learning (ML) and deep learning (DL) applications of all types. Here’s why.
Back in the early 2000s, media and entertainment companies began to require large-scale shared storage to handle the increasing frame resolution demanded by viewers and to support a proliferation of media formats. Technology has progressed to accommodate the increasing demands of these workloads, and the result is advanced storage like Dell EMC Isilon.
What’s exciting is that the same technology used to accelerate video processing can also serve as the critical foundation for modern data analytics applications. You can’t have Big Data if you don’t have Big Storage. But larger storage simply isn’t enough to deliver performance at scale.
Let’s look at the financial trading industry. Today’s trading firms are transitioning their algorithmic models from intraday to multi-day trading. This, coupled with ongoing exponential growth in daily transactions, means that financial systems can no longer store their active trade data sets (> 10 TB) in memory. Without advanced storage, firms would need to reduce the data sets they are working with, give up near real-time performance, or limit the number of simultaneous processes (e.g. concurrency).
This isn’t just a problem for financial trading. Utility companies need to work with increasingly greater data flows as more and more meters are brought online. Massive numbers of smart sensors are being deployed in Industrial 4.0 applications, smart buildings, and smart cities. There is more data to collect, and as a result there is more data to process. If this data is going to be useful, it must be processed in as close to real-time as possible. After all, it’s pointless to learn that a stock is a good buy or that a system is about to fail after the window of opportunity to act is past.
What’s extraordinary about Dell EMC Isilon is how it overcomes the limitations of previous memory architectures. HSM automates movement of data between high-speed (i.e., high-cost) storage and lower-speed storage to balance cost with access speed. HSM requires massively parallel I/O to achieve this and avoid bottlenecking the system.
With up to 945 GB/s and 17M IOPS, Isilon provides industry-leading storage efficiency of 85 percent. Isilon’s outstanding performance crosses the board, as shown by its impressive STAC benchmarks. When combined with Dell EMC PowerEdge servers and Nvidia GPU technology, the results are near real-time with high currency for even the most demanding AI and ML applications.
Two other important factors are scalability and transparency. Isilon scales from 10s of TB to 10s of PB. In addition, HSM is handled transparently. With these capabilities, Isilon is able to serve as a Data Lake for running analytics, enabling developers to focus on what to do with data rather than expend their resources trying to manage it.
The benefits to financial trading firms are significant. They can train, validate, and score models faster, reliably share access to a single copy of data, and deploy more advanced and iterative models. Enterprise security and compliance becomes easier to manage. And higher model accuracy directly impacts the bottom line. For example, one large New York city hedge fund has been using Isilon storage since 2007 and is now able to leverage 30.5PB of tick data analytics in their daily operations. They have also seen a significant performance increase (390% more) in throughput, allowing them to run larger, more sophisticated research jobs without any impact to performance. In addition, as the company grows, they are able to scale storage much faster than compute resources.
Data analytics for applications like AI and ML are steadily becoming more important, and storage is the bridge to get there. With the throughput, scalability, and transparency of innovative storage technology like Isilon, there are no bottlenecks to innovation.
Learn more about how to eliminate storage bottlenecks for algorithmic trading and other modern data analytics applications in this webinar Financial Quantitative Research in a Hybrid Cloud World or check out this Tick Data Analytics white paper. You can also register for a STAC summit near you to know more about how storage technology continues to evolve.
