Big data is a big word that’s made a big time in business circles. Initially, it was the big fish who used it, but with the proliferation of AI-based tech in this decade, it’s becoming more accessible industry by industry. As of 2023, around 59.5% of organizations, no matter the industry, used data analytics (approaching what we call big data) – and transportation and logistics is one of the pioneering domains. The global transportation analytics market was deemed to be somewhere at $18.58 billion in 2023, including route management, fuel management, and predictive maintenance. It’s no small feat to implement big data in transportation, though.
This involves changing the digital culture, in some cases, the entire modus operandi for several departments (and especially in between them) – but the ROI is worth it. About 92% of organizations are reporting a positive impact from data and analytics investments, and going for big data is going for the jackpot in many cases. Let’s explore what this looks like in transportation and logistics – but first start with the definition.
What is Big Data?
There’s a difference between a traditional data-based approach and big data. In a nutshell, traditional data analytics:
- is fed standardized datasets
- can handle certain amounts of data (otherwise the system “chokes”)
- relies on batch processing: you upload the data, then it’s processed.
What sets big data apart is the 3V approach: Volume, Velocity, Variety. Big data relies on technologies like Hadoop and NoSQL databases to handle terabytes and petabytes of data in a whiff. Big data also involves handling real-time (or near-real-time) updates in data to enable faster decision making.
There’s also the structure of the data: with the traditional approaches, you are expected to make the data digestible to the system in a single format. Essentially, it means creating something like a 3-dimensional, maybe 4-dimensional, 5-6-7-dimensional, n-dimensional table for the machine to “eat up” and then giving the machine a preconceived hypothesis to test. Big data systems are omnivores with intelligence, though: they discover patterns from complex datasets, without the need to curate data, which means they can “eat” XML files or even social media posts and TikTok videos.
In brief, while traditional data analytics approaches were but an apprentice, big data has become a master.
Transportation Data Sources and Types
So one of the big advantages of big data approaches is that they can work with different types of data to create hypotheses and test them. The “types” in question, then, are not just the different formats but the different sources of data. As of now, the transportation sector sits on a data goldmine that includes:
- Connected Vehicle data: real-time information from vehicles: vehicle location, speed, acceleration, and even internal diagnostics, meaning you can know much about traffic conditions and vehicle performance.
- GPS data: the driver’s probably got a smartphone; plus, there are navigation systems and fleet management tools. GPS data provides precise location and movement information.
- Traffic sensors: inductive loops, cameras, and radar sensors that monitor traffic flow, speed, and vehicle counts on roadways.
- Public transit data (in urban communities): information from ticketing systems, vehicle trackers, and passenger counters helps optimize public transportation services.
- Weather data, etc.
This is not the entire list, though; big data in transportation can also potentially use things like social media data for sentiment analysis (e.g. traffic jams) or many more.
Big Data and Transportation Analytics Use Cases
Getting more industry-specific, here are some of the most promising use cases of big data in transportation based on what we’ve seen in the last decade:
#1 Route Optimization
Route optimization often revolves around things like traffic and weather patterns, and that’s where big data is the true heir of the synoptic’s work (only it’s even more precise). For example, UPS has its ORION system, analyzing delivery data to optimize the routes that the drivers must take, saving the company whole gallons of fuel (not counting the millions of miles paid per hour) – all based on the traffic and weather conditions.
#2 Predictive Maintenance
Like any other equipment, vehicles tend to deteriorate from use – and when they do, it’s an issue. With big data plus sensors, you can predict when the infrastructure will likely require maintenance, so that it’s no longer a big surprise (and an unpleasant one at that). This is even used by airlines like Delta. They have predictive maintenance systems in place to reduce delays and cancellations by monitoring airplane engine data. They analyze sensor data in real time to predict when parts need replacement.
#3 Supply Chain and Freight Management
It’s one thing to have the products in the warehouse; quite another, to deliver them to the end consumer. This stage relies on a balance between the demand and the inventory levels, Amazon uses big data analytics to optimize its vast logistics network, improving its inventory management and ensuring faster deliveries. Real-time data allows the company to make quick adjustments to shipping routes and stock levels.
#4 Traffic Management and Congestion Reduction
Big cities are the natural points in any transportation network, but they present more challenges than a highway. This is where companies and city authorities work side by side to manage the bottlenecks – but private enterprises can also work to mitigate the challenge on their own. However, city conditions (especially in the metropolis) can pose its own requirements.
Barcelona is a good example. The city’s transportation system uses big data to control traffic signals and manage bus routes in real time, reducing traffic congestion (something that Lionwood has replicated with its own project) – and that is also beneficial for businesses, since not all transportation routes can avoid big cities.
#5 Enhanced Safety and Accident Prevention
Road safety is another vital (literally vital) factor in transportation. What big data can do here is analyze accidents, driver behavior, and environmental factors to reduce accidents, e.g. help adjust speed limits. Vision Zero is a good example here, as applicable to NYC: by analyzing the data mentioned above, they managed to reduce accidents on the road – and there are multiple counterparts to their effort.
A Note on Urban Transportation
What’s really interesting in the transportation domain is how it’s connected to the issue of urban mobility. Big data is a game changer here, as well, utilizing data from GPS devices, traffic cameras, and connected vehicles. In most of the initiatives here, public transit systems are the primary beneficiaries, enabling agencies to adjust schedules, routes, and capacity to meet demand more efficiently. Real-time tracking data also enables accurate arrival predictions for passengers, enhancing the overall transit experience.
Key Considerations
Big Data is one of those trends in transportation that promise to become mainstream (and and gradually becoming it). The ROI seems reasonable, too, especially with route planning, where the average savings on operational costs amount to 15-20%. However, there are several considerations to have in mind before plunging into adoption. The good news is that the very companies that implement big data solutions for transportation can also provide consultation on how to prepare for it. The attention points are as follows:
- Data quality + integrations: transportation data comes from diverse sources, including IoT devices, mobile apps, and legacy systems. Even though big data approaches can work miracles on diverse sources, an even better pathway is for someone to ensure the data is as complete as it can be, balancing tech capacities with economical feasibility
- Data security and privacy: there are lots of data that include sensitive details that might be protected under various legislations: travel patterns, location, etc. This is why it’s important to ensure the collection of data is compliant to things like CCPA, GDPR, and other similar regulations.
- IT infrastructure (and scalability): since everything’s on the move, transportation means continuous data updates. That means the solutions used to handle the resulting big data need to be scalable, these days typically achieved with cloud computing.
- Employee recalcitrance: there may be skill gaps as well – the existing staff is not always willing to work in the new way. This is already a solved problem, though: while some organizations may need a data engineer in place, most will do with a training course.
How to Start Using Big Data for Transportation
Is it difficult to start using big data? Depends on how hard it is to define the strategic needs at the point of implementation. As with most data projects, there needs to be a solid, well-defined, transparent understanding of what specific problems need to be solved, e.g. “we need to reduce traffic congestion [optimize routes/something else] by [doing this or that] based on [this or that data]”.
Once the objectives are defined, the company and implementation team can go through the next stages:
- Assessing the current data assets and sources and working to fill in the possible gaps
- Establishing the tech infrastructure – cloud platforms, data storage, analytics engines, etc.
- Engaging the necessary talent in the team – preferably both data specialists and industry experts.
- Developing a data management and governance procedure (plus addressing privacy concerns)
- Launching a pilot project or PoC on a smaller scale (e.g. optimizing routes on a particular line or in a particular area).
Another necessary line of work is instilling a data-driven culture within the organization, so that the actual decision making is really taking advantage of big data regardless of how automated it is.
A skilled team of experts that are competent in working with transportation can help at every stage of this process. The main point here is that the big data solution you end up developing is perfectly aligned with industry best practices and the individual goals of your organization, and previous experience with the industry helps the implementation team suggest the optimal solutions. For Lionwood.software, transportation and logistics have long been one of the principal industries to cater to, so you can consult our experts about your big data project at any time.
