It’s been quite a year in the storage space. This blog post covers everything you need to remember from 2018 about the emerging storage trends and what you can expect to see on your radar for 2019.
- Machine Learning
- Security: GDPR
- Cloud Adoption
- Multi-Cloud Orchestration
- Storage Class Memory
1. Machine Learning
Machine learning (ML) allows for automation of analytical models for data analysis. It is a branch of AI that focuses on creating systems that can learn, identify and decide. While ML has several applications, from self-driving cars to speech recognition, there are significant advantages to data center optimization.
The year 2018 was big for AI and ML. Waymo, Alexa, you name it. Gartner predicts that
40% of data science will be automated by 2020. ML can enable software to dynamically balance and optimize data movement to reduce wear-out. ML can expand on its own learning as it operates in its environment, thereby using ML algorithms in storage solutions to perform tiering, which can significantly boost efficiency and performance to previously unseen standards. With the promise of several benefits of ML in data centers, Deloitte Global predicts that the number of pilots and implementations will double in the coming years. Further, AI spending is expected to reach $60B by 2021.
Forrester called 2018 the “Year of Enterprise DevOps.” While DevOps has existed for quite a while, there are still many barriers or hurdles to get executive buy-in. But 2018 has seen a lot of positives for the progression of DevOps. And the past year has been big for CI/CD. CI is a DevOps pipeline that requires teams to implement smaller changes more frequently, but with checks in the code with version control repositories. Therefore, there is a lot more consistency in the building, packing, and testing of apps, leading to better collaboration and software quality. CD begins at the tail of CI. Since teams work on several environments (prod, dev, testing, etc), the role of CD is to automate code deployment to these environments and execute service calls to DBs and servers.
CI/CD has been referenced in the Agile Manifesto, which was created back in 2001, but only now do we have the right tools coming to the mainstream to fully reap the benefits of CI/CD. In 2018, we saw the adoption of CI/CD and the use of container technologies by many organizations. But one of the challenges is the handoff to production. 2019 will be the year where “jobs-as-code” will help DevOps teams avoid the manual barrier of moving to production.
Cloud insecurity has been increasing in 2018 and is expected to grow even more in 2019. Ransomware is starting to fade away, as criminals are shifting to other methods, but it will still remain a problem. In May 2018, the General Data Protection Regulation (GDPR) took effect in the EU, and this led a lot of organizations, even those operating outside the EU, to become compliant with the policy. While the penalties of non-compliance didn’t happen in 2018, they are highly likely to occur in 2019. Organizations have been looking toward storage solutions that have security and data protection baked into the system by default. This eliminates the need to invest in a system for security. Also, AI is starting to be of a great advantage in security. Machine learning–based malware detection is a model that security companies are working on, and 2019 is a year where we might see some of their offerings in this area.
4. Cloud Adoption
Organizations are continuing to invest in cloud providers. However, regulations and security concerns are the main reasons that certain organizations are staying away from public cloud providers. Gartner forecasts that 2019 will see 17% growth in cloud adoption, which will reach a whopping $206B in revenue. North America is forecasted to lead cloud storage technology over the next five years, while the Middle East, Asia Pacific, Latin America, and Africa are expected to see an increase in usage from improved network availability.
Financial services have the highest percentage of server images deployed in private or public clouds, and they’re way ahead of the healthcare and insurance industries.
5. Hyperconverged Infrastructure (HCI)
Hyperconvergence was the buzzword in 2017 and was expected to grow in 2018. However, it turns out there is less enthusiasm for hyperconverged infrastructure. HCI seems to bear some inefficiencies when the environments scale and is not very flexible to easily scale compute and storage separately. Organizations are starting to rule out HCI since it does not integrate with existing systems and results in vendor lock-in. According to this survey, 42% of the decision makers consider vendor lock-in their top concern. HCI technology may not be mature yet, and we’ll find out how it shapes up at the end of 2019.
Software-defined storage (SDS) started gaining popularity sometime in 2013. SDS has been attracting organizations for two main benefits: flexibility and cost. Unlike HCI, SDS allows integration of multiple vendor storage systems under a single pane of glass, saving time and resources for IT. SDS is a policy-based data storage provisioning and management software layer independent of the underlying hardware. It is hardware agnostic and enables high performance and high flexibility while lowering costs. By 2021, the SDS market is projected to grow at a compound annual rate of 37% to a whopping $22.56 billion.
By the second quarter of 2018, most major enterprise storage vendors had announced an all-non-volatile memory express (NVMe) version of their storage arrays. IDC predicts that NVMe over Fabric (NVMe-oF) will drive more than 50% of primary storage revenue. NVMe was designed to employ the PCIe transport, which is the communication channel between CPU and peripheral devices. It has low latency and high bandwidth, which is ideal for storage, but PCIe is predominantly used in servers. NVMe-oF allows the NVMe protocol to be mapped over fabric, allowing storage nodes containing NVMe SSDs using PCIe to connect. This results in incredibly higher performance and low latencies, as it uses a storage protocol that leverages parallel and low latency data paths. IDC expects that by 2020, NVMe enterprise device revenues will reach 20% of total enterprise storage device revenues.
8. Multi-Cloud Orchestration
Resiliency and flexibility of the multi-cloud model have led IT teams to move workloads across clouds. It can be tedious to set up the orchestration and automation, but it is achievable. Multi-cloud deployments are gaining popularity and will continue to see exponential growth in the coming years. According to The Future of the Data Center in the Cloud Era, a report by Gartner, it is expected that 70% of enterprises will implement a multi-cloud strategy in 2019. Multi-cloud comes with some challenges, as it can increase the overhead cost from segregating the workloads across multiple providers. It also requires a governance process in place and has to support different environments. However, the deployment of containers and tools like Kubernetes for container orchestration, which can run the code on any cloud provider, is paving the way for true multi-cloud deployment and allowing businesses to reap the multiple benefits of multi-cloud solutions.
Containers are changing the way apps are being deployed. DevOps are turning toward containerized environments and microservices–based architectures, as virtualization isn’t keeping pace. Containers are what virtualization was in 2002 and what the cloud was in 2010. In the very near future, containers will become the go-to option, as containers are getting closer to mainstream adoption. Containers take their structural makeup by removing the inefficiencies in the traditional architecture. They are enclosed units of a cloud environment, comprising all that is required for code to run in isolation. The isolation boundary encapsulates the application and its dependencies, while the OS is abstracted on top of hardware virtualization, which makes the app highly efficient and scalable. Kubernetes is the leader in container orchestration, with Docker Swarm and Apache Mesos following. The container market is expected to go from $762M in 2018 to $2.7B in 2020. 2019 will be a big year for the evolution of containers from centralized to distributed, with container security and governance starting to gain focus.
10. Persistent Storage Class Memory
Persistent storage is still the biggest challenge for businesses using containers. While a containerized infrastructure provides several benefits, the main barriers that teams face are related to integrating it with their existing infrastructure and making it useful for stateful applications. The original concept of containers didn’t address the need for persistent storage; they were “on-demand” boxes that would come to existence, run the code, and cease to exist. This worked quite well for stateless applications by making use of temporary storage. However, for prolonged use of containers, there has to be a storage handler. While there are a few available solutions, such as Amazon Elastic Block Store (EBS), Kuberenetes PersistentVolumeClaim, Docker data volumes, that are starting to address this challenge, 2019 will be the year when you see a few companies lead the market with their own offerings for this use case.
Learn about Cloud Native Storage for Stateful Containers here.