2024 trends in data storage and architecture

1: Data centres will undergo sustainable renovations and retrofits

“Data centres are under pressure to become more energy-efficient than ever — AI-compute will continue to pile on the pressure. The GPUs that power AI at scale require radically different infrastructure from traditional CPU-powered compute. Many operators that want to embrace AI-compute must overhaul existing power, cooling, and cabling infrastructure.

“For many data centres, this entails expensive renovations and retrofits well before the site’s end of life. If operators want to avoid large-scale changes before a site’s planned end-of-life, they must adopt a holistic design approach that facilitates interchangeability in parts and site functionality.”

Niklas Lindqvist, Nordic general manager, Onnec 

“Nearly 90% of data in cloud data centres resides on hard drives. As data centre infrastructure approach natural refresh cycles, there is an opportunity to replace fleets of lower-capacity hard drives with higher capacity drives.

Today, the average hard drive capacity is 16TB, based on conventional PMR technology. Because it can physically store more bits on a single drive, new HAMR technology will enable data centre managers to populate those same slots with 30TB high-density drives, yielding significant power and space savings and massive TCO efficiencies — including CPU, RAM, and floor space.

“HAMR technology will also help data centre operators to further reduce their carbon footprint with more durable and energy efficient storage, playing an even more influential part in tackling the 53.6 million metric tons that make up the global e-waste mountain.”

BS Teh, chief commercial officer, Seagate Technology 

“Many organisations that are reliant on datacentres are reporting that their most pressing issue right now is one of capacity. A growing number of datacentres are full, and don’t have the space or power available to deploy new platforms.

“In 2024, this will result in widespread efforts to achieve efficiency gains, even on existing datacentre platforms, as this is the only way they will be able to reclaim space and power to accommodate the use of new technologies inside the datacentre.

“To optimise the sustainability of existing datacentre footprints, we’ll see operators looking to switch to new, more power efficient technology, with smaller space and cooling requirements. This is in essence extending the life of the datacentre – an essential factor when considering the need for new technologies in the wake of the rise of AI.”

Patrick Smith, Field CTO for EMEA, Pure Storage 

2: The rise of MLOps

“Demand for machine learning engineering roles has been steadily rising, with a more recent trend of hiring managers seeking machine learning operations (MLOps) engineers — professionals who can bridge the gap between data scientists and data engineers. By sitting within and working alongside both teams, these people purely focus on building, deploying and maintaining the various platforms and pipelines to facilitate the work of data scientists.

“Some companies are even going as far as building brand new MLOps teams, but it’s a very new area. Many companies haven’t yet developed that function, and where those people sit within the business is still quite undefined. The roles that we’ve placed have typically been working alongside a data science and a data engineering team.

“With an exponentially increasing recognition of the value of data and the development of a range of new tools, we’re likely to see many more types of organisations looking to hire ML specialists to leverage their data for the benefits of the business.”

Rosie O’Callaghan, business manager, Data Science and Machine Learning, Harnham

3: Businesses will rethink data architectures

“In 2023, organisations started to experiment with data lakehouse capabilities, but its true value hasn’t shone through. So, 2024 will be the year of the open data lakehouse whereby all the elements of an organisation’s tech stack start to distribute data between services more effectively. This will enable concrete, real world use cases – such as taking a point-in-time snapshot of data for financial compliance – as organisations start to take control of their data and treat it more like code.”

Chris Royles, EMEA field CTO at Cloudera 

“Organisations are going to break down silos between different data types with unified data storage — which combines all a company’s data types across on-premises and cloud environments—to fuel the new era of AI and analytics.

“As a result, we are going to see more organisations rethink their data architectures to consolidate data streams and treat them as a single source. Unified data storage will help make enterprise data more easily accessible and unlock hidden connections between different types of data.”

Matt Asay, VP of Developer Relations, MongoDB

4: Specialisation of job roles

“Data science as a specific subject of study has become increasingly prevalent in recent years. Numerous universities now offer data science master’s degrees and some of the country’s top universities such as UCL, Edinburgh and Kings College London now offer courses in machine learning, data science, and AI.

“Unsurprisingly, this is having a knock-on effect on the candidate make-up of the market. We’re seeing more people coming through with specialist degrees. Many may have chosen to complete a bachelor’s in a STEM subject such as maths or physics and have then taken a specialist master’s degree or PhD. For an employer, someone who holds a maths degree and a machine learning masters presents an ideal candidate.

“We’re likely to see even more specialisation as the role AI can play across the business domain becomes clearer and better understood by stakeholders.”

Rosie O’Callaghan, business manager, Harnham

To read this year’s AI predictions click here