Mastering the right technology for storing data in petabytes as management and analysis of data is posing to be a huge challenge.
12 Oct 2018 | 4336 Views | By Jim Simon Sr, Quantum
The automotive industry relies strongly on digitalisation. If connected cars and autonomous driving are to become a reality, each car will produce data in petabyte dimensions each year. This makes backup, storage, management and analysis of data an enormous challenge — which can be mastered with the right technologies.
This is an Autocar Pro Plus article. Subscribe to continue reading.
The automotive industry relies strongly on digitalisation. If connected cars and autonomous driving are to become a reality, each car will produce data in petabyte dimensions each year. This makes backup, storage, management and analysis of data an enormous challenge — which can be mastered with the right technologies.
The future of the car raises many questions. Today's mobility systems are confronted with various challenges. Globalisation and urbanisation are driving up the volume of traffic enormously. This is why car manufacturers and technology providers are working to make driving more efficient, safer and environmentally friendly. The key factor is digitalisation – both of the car itself and its surroundings. It is certain that autonomous and connected driving will soon become a reality.
Manufacturers have already proven the general suitability of numerous prototypes of autonomous and connected cars, trucks and buses for traffic. While the first autonomous buses and taxis are already driving on test tracks through Shanghai and Zurich, a German pilot project researching autonomous cars in real-world operation has been taking place for around two years on the A9 motorway between Ingolstadt and Munich.
Quantum DXi4700 serves as the core of data protection solutions for small- to medium-sized businesses and remote site protection for enterprise projects. It provides high performance and capacity-on-demand scalability.
Driven by the rapid technological development in the fields of software, artificial intelligence, microchips and sensors, autonomous and connected vehicles are in the fast lane towards mass production.
Large construction site: digitalisation
There is still a lot of work to be done before autonomous vehicles are a normality: from a reorganisation of the transport infrastructure, to the creation of an adequate legal framework, as well as IT security and data protection measures. In addition, there is another quantifiable challenge: the new volume of data. By reinventing itself as a digital pioneer in comparison to other industries, the automotive and transport industry automatically produces a lot of data.
A test vehicle already generates between five and 10 terabytes of data per day. This results in a petabyte-sized amount of data per year – for a single vehicle only. Automobile manufacturers do not use just one, but many test vehicles, so this calculation has to be multiplied accordingly. If autonomous and connected cars are to be the new normal on our streets one day, this will generate a much larger amount of data. After all, the car of the future is nothing more than a rolling computer connected to the environment, infrastructure and the Internet. And this computer continuously produces and processes data in order to be usable as a means of transport on wheels.
With the automatic emergency call ‘E-Call’ becoming mandatory in the EU from March 2018, practically every car produced is already a connected car. The automotive future, therefore, depends on questions like how the new data dimensions will be handled, how data can be managed intelligently, efficiently and safely.
Data collapse preprogrammed?
Finding the right answer to these questions is not easy. After all, it is not only the end product itself – the connected car – that produces a huge amount of data, but an entire industry. Automakers work in a complex global network of suppliers, service providers and partners, which all have to build up an adequate internal IT infrastructure in order to efficiently maintain and manage their critical company data.
In addition, with the right technological equipment, they have to be able to achieve the necessary degree of flexibility for the exchange of data between the individual parties. This is a data exchange that can take years from R&D to production, product launch and distribution. The connected car follows in the last step, whose data has to be managed in an intelligent interaction of all components, from the on-board computer to the data centre and in the cloud.
Who is responsible for a car accident?
The choice of technology solutions and the strategy for securing, exchanging and long-term archiving of data for car manufacturers is critical to success and legally important from the outset. This can be demonstrated using the example of a car accident.
In addition to traditional backup and intelligent data management, it is essential for OEMs to evaluate this data so that the car can make smarter decisions.
Even then, in theory, car manufacturers can still be held responsible. In the event of legal proceedings, they are obliged to prove how the vehicle has been tested and why it has been classified as roadworthy. They must make this plausible on the basis of the test data, which must still be available to legislators after seven to ten years. In case of doubt, the accident can then be attributed to human negligence and the manufacturer cannot be held responsible. Much of this test data is generated in subsystems by suppliers who also maintain this data for their own protection. However, it is essential that they are available to manufacturers. Basically, everyone in the supply chain is obliged to secure data and demonstrate a conscientious approach to research, development and production of the automobile. Legal issues relating to the development of connected and autonomous cars are becoming more explosive, with the main question being: Who is to blame for an accident involving an autonomous car – man or machine?
Playing catch up in strategic data management
The example of the car accident illustrates in all its legal pitfalls that no company in the automotive industry can afford to neglect the subject of strategic data management. And yet, as studies have shown, there seems to be a gap between claim and reality. While the majority of OEMs and suppliers want to achieve a very high level of digitisation, just one in five of them have a strategy for digitisation and data management. This is a paradoxical situation, considering that companies have remarkable IT solutions at their disposal, which are able to relieve them of a lot of work.
Proven and useful: the ‘3-2-1’ rule
The IT infrastructure of global automotive manufacturers is usually highly complex and has grown over decades. Manufacturers still regularly find themselves trapped in traditional silo structures, which prevent cost-effective, powerful and agile data management. And this starts with intelligent data protection, which will be all the more important given the future of data in petabyte dimensions. It is therefore advisable to continue to back up data according to the proven ‘3-2-1’ rule, where at least three copies of data are stored on two different backup media while one extra backup is held off-site. Even in the era of the mobile and multi-cloud, this is still the optimal method against hardware failures, cyber-attacks or data corruption and data loss caused by the ‘human error’, as it is known within the IT industry.
Effective and cost-effective: tiered storage
Powerful back-up media such as high-performance disk or solid-state disk with data deduplication can be used to speed up the initial backup and recovery of data. This feature automatically searches for, and removes duplicate data, without compromising its integrity. For longer-term retention such as cold or active archiving of data, which does not have to be accessed in real time, cost-effective storage media such as tape object storage or cloud are available. In addition, it is critical that data is not corrupted or deleted. When choosing the storage medium, IT-practitioners should always proceed according to the ‘tiered storage’ principle. With ‘tiered storage’, data is distributed to different storage media according to different categories, such as access rate or vulnerability. The goal: reduction of overall IT costs with high performance access to active data. For example, mission-critical, highly frequented data can be stored on expensive disk or SSD, while less frequently used data can be distributed to lower-cost storage media such as tape or the cloud.
Rule-based and transparent: categorisation of data with software
It is obvious that automating the categorisation of data can be highly beneficial. As a result, technology providers offer software that can automate this complex data management task based on predefined rules. A valuable benefit of some tiered storage solutions with rule-based data management is transparency. Even if a file has been automatically moved to another storage level, it still appears to the user in its usual place. To choose the right data management software, it’s important to find one that fits seamlessly into collaborative environments, a critical aspect in the R&D of new car models. Even in a storage environment with several hundred petabytes of data on disk, tape or cloud, it should not lose any of its performance. One absolute necessity is to ensure it has detailed monitoring, alerting and scan/search capabilities. For example, the data management software Quantum StorNext was developed especially for demanding workloads in complex IT environments.
Scalable and powerful: Scale-Out NAS
Specialised storage solutions must have enterprise NAS (Network Attached Storage) because of the increasing amount of data acquisition, processing, analysis, distribution and archiving. NAS refers to a networked, easy-to-manage file server. However, traditional enterprise NAS cannot keep up with many IoT scenarios from connected cars to autonomous driving. For this reason, many users have opted for the so-called Scale-Out NAS as an alternative over the last 10 years. The big advantage of Scale-Out-NAS is that the entire amount of hard disk space can be expanded flexibly if required. This in turn leads to benefits such as lower costs and more efficient utilisation of hardware resources. Previously, Scale-Out-NAS has often lacked scalability options and ways of integrating cloud strategies or sharing data in the past.
Quantum is now addressing these problems with its industry-first storage solution, Xcellis Scale-Out NAS, which combines the critical data management capabilities of Enterprise Scale-Out NAS with cost-effective scaling options. In the solution, this happens through clusters, a number of networked servers that can scale together or independently to reach hundreds of petabytes of capacity and terabytes per second of performance.
Urgently needed: storage platforms for the 5G age
The Achilles' heel of the autonomous and connected car is the network through which it communicates with other vehicles and the infrastructure in the data centre. For this reason, this year's Consumer Electronics Show focused on the new 5G mobile communications standard, which promises high speed, more bandwidth and extremely low latency. Fully integrated end-to-end storage platforms that can cover the entire lifecycle of data are required to give the best possible error-free auto communication and data processing for tasks like backup and analysis. With 5G speeds of 10Gb per second and latencies of less than 10 milliseconds expected, integration with the most powerful Ethernet networks, for example via SMB and NFS interfaces, should be considered for storage solutions.
There is no way around AI
Even though the future of the car still raises many questions today, it will definitely be a data-intensive and software-driven future in which artificial intelligence will be indispensable. In addition to traditional backup and intelligent data management, it is essential to evaluate this data so that the car can make smarter decisions. For this reason, the future belongs above all to storage solutions that integrate well with artificial intelligence to query data across several axes in order to uncover events, objects, faces and words.
Data management is key
Even if the majority of cars today still run on petrol or diesel, the motor of transformation towards connected and autonomous electric cars is now running at full speed. Automobile manufacturers therefore cannot afford to avoid investing in their data management strategy. And that's what Dinesh Paliwal, head of the American electronics company and automotive supplier Harman, said: "Data is the future currency in this industry – and anyone in control of data management will be the king."
(This article was first published in the 1 June 2018 issue of Autocar Professional)