Embedded Hardware Advances while Software Practices Lag

It amazes me how quickly innovations in semiconductor technology and hardware design continue to advance and support increasing levels of integration and performance. Hardware design, verification, manufacturing technology, and tools and processes are always moving forward to re-invoke Moore's Law. When I read about single-board computers already shipping with dual-core Intel® Xeon® processors with Hyperthreading technology or new ATCA-compliant blade servers, I think about these advancements and the opportunities for more software functionality. The growing complexity of integrating and verifying more software will be the main challenge going forward.

I recall a quote from last July's Design Automation Conference: "It's the software, stupid!" The point is that software will continue to be the long pole for delivering embedded products. The technology, processes, and tools to cost effectively produce a multi-core, Intel Xeon processor-based, single-board computer are already in place. But the technology, processes, and tools to effectively integrate and verify the embedded software are woefully stuck in time.

Embedded-software verification methodologies and practices haven't changed significantly over the last two hardware generations. At one time, single-board computers had only a general-purpose central processing unit (CPU). They then upgraded to highly integrated systems-on-a-chip (SoCs) or network processors and now multi-core processors. During this evolution, the corresponding embedded software has increased exponentially. But engineering teams haven't evolved their software testing processes to effectively manage the new scale and complexity of embedded-software development.

Today, embedded-software development involves a global supply chain of engineers, offshore outsourcers, open-source, and third-party software vendors. All of them are delivering hundreds of software components. In a recurring pattern, software development generally appears to progress smoothly through the implementation phase. From that point on, however, projects enter long, churning, and unpredictable integration and system test phases. This pattern indicates that companies are really struggling to integrate and verify so much software from so many sources. As a short-term fix, software managers add more engineers and resources to shore up integration and testing efforts. Yet this approach has limited effectiveness and very high cost. Plus, software releases are still often delivered late and with compromised quality.

Today, integration and testing are already consuming a disproportionate amount of time and resources in the development cycle. Compared to software implementation, it isn't unusual for integration and system-test efforts to take 5 to 10 times as long and be staffed with very large, growing teams. Plus, too many defects are escaping to the late stages of development. These defects are wreaking operational havoc on software teams while making software delivery agonizingly unpredictable. Without achieving the fundamentals of a scalable, reliable, and predictable software process, software verification and integration challenges will only worsen with the new generation of embedded hardware.

For example, how might embedded-software development be impacted by multi-core processors—perhaps with cores that are tailored for specific code (i.e., one for applications, another for encryption, a packet processor, etc.)? Multi-core processors might force development teams to be more disciplined in their overall software design. In addition, specific types of software will be executed on certain purpose-built cores. As a result, more cleanly architected software design and code might be forced to be more componentized with well-defined interfaces between major functional areas.

Yet the software will continue to be an enterprise-scale effort that leverages resources all over the world. Although the volume of code that's required to enable and deliver all of the various features would largely be the same, there would now be an additional dimension of software complexity from multi-threading or concurrent threading. Ultimately, all of the various software components would still need to be integrated, tested, and verified to operate properly as a cohesive whole. If the current methods for integrating and testing complex embedded software are already breaking down, another level of software complexity would certainly exacerbate the problem.

How should embedded development practices change to effectively manage the growing scale and complexity of building software—especially when it involves a global software supply chain? Semiconductor and hardware companies have long understood the need to align the various players in the design and manufacturing supply chain as well as to automate as many processes—especially verification. Likewise, companies need to adopt collaborative practices that unify the embedded-software supply chain while enabling the wide-scale automation of software testing and integration.

This new approach would require all embedded-software suppliers—both internal and external—to create and deliver reusable automated tests with their software. New processes would then leverage, reuse, and apply the automated tests as much as possible. In doing so, they would continually verify software at various stages and levels (by individual developers, by functional teams, on fully integrated engineering builds, hourly, daily, etc.). With automated reporting, the processes also would provide comprehensive, quantitative metrics and visibility into quality and completeness at various stages of development.

Implementing this approach certainly requires a change in mentality. The diversity of embedded-software tends to breed a culture of individualistic software-development practices. For instance, a developer who implements and tests IP-routing protocols would do things differently from an engineer who's creating an embedded database or device drivers. Yet it can and has been done successfully and with big payoffs. The resulting software-development process produces higher-quality code earlier and with fewer resources. At the same time, it provides better visibility and predictability into software health and completeness.

The latest single-board computers certainly feature very exciting and compelling semiconductor and hardware advancements. Without scalable, unified software-verification processes, however, engineering organizations will continue to struggle through software development. They won't have the time and proper mindset to truly leverage the full potential of these hardware platforms.

Mark Underseth is CEO, President, and Founder of S2 Technologies. He has over 20 years of experience in the design of embedded communication devices including mobile-satellite, automotive, and digital-cellular products. Underseth received a BA and MS in Computer Science. He has been awarded three patents. In addition, Underseth currently has five patents pending in the area of embedded systems.