Multi-Core Software Debugging Challenges Traditional Approach

Embedded designers must move beyond the "fix-one-bug-at-a-time' methodology when debugging software written for multi-core systems.

By Doug Gaff

Anyone who develops software is intimately familiar with the traditional debugging process: Start a debug session, stop the application at a breakpoint, step through code, examine data, fix bugs, build, and repeat. All software-development environments provide this basic set of capabilities. Yet this cycle and many of the tools that implement it inherently focus on debugging one thing at a time.

Enter multi-core. The term “multi-core” typically refers to a single chip containing more than one logical device capable of executing code (e.g., multiple processors, a processor and a DSP, a processor and an FPGA). It also can mean multiple processors on a single target board. For companies that build development tools, multi-core should theoretically be an extension of a known use case. After all, device software-development tools already deal with lots of “multiples:” targets, processors, processes, threads, operating systems, debug interfaces, debug engines, and sometimes even multiple programming languages.

In reality, mixing so many fundamentally distinct “debuggable entities” in a single development tool can be very difficult. Different operating systems often require different debug engines, such as GDB or vendor-proprietary. In addition, they often demand different debug connections including network or JTAG. Some debug engines can only support specific processor variants. In many cases, the debuggers that are designed for bare-metal access to chips via debug hardware aren’t designed to debug applications using an operating-system debug agent. As embedded devices become more feature-rich, so do the collection of tools required to develop and debug the software running on those devices.


Common Framework
To address this complexity, a common framework is needed to enable interoperability between multiple development solutions from various vendors. The good news for both device software developers and tool vendors is that there already exists such a framework: Eclipse (www.eclipse.org). Eclipse is both an open-source community and a collection of open-source software projects that provide everything from Java development to software test infrastructure and business-report generation. Most importantly, the overarching mission of Eclipse is to create frameworks and tools that allow companies to build commercial products and solutions on top of the Eclipse technology.

The Eclipse technology has been adapted to support virtually all development languages in use today including Java, C/C++, Perl, Ruby, Fortran, and even COBOL. Perhaps the two best-known language projects in Eclipse are the Java development tools (JDTs) and C/C++ development tools (CDTs). Each of these projects contains a wealth of capabilities to support the edit-compile-debug cycle mentioned previously: an editor with code highlighting and completion, code browsing and refactoring, managed builds, and debug capabilities.

Most relevant to the subject of multi-core debugging, however, is that these and other language-development projects build their debuggers on a framework called the Eclipse Platform Debug Model. This framework provides the generic building blocks for a debugger: debugger views like memory, registers, and expressions; a simple debug hierarchy that shows a process, thread, and stack frame; run control actions for starting, stopping, stepping, and breakpoints; and a launch mechanism for initiating a debugging session. This framework is designed to be extended for use with a variety of debug engines.

Yet several device software-development companies that build commercial products on top of Eclipse have discovered the need for additional customization of this framework. These companies formed an Eclipse project called Device Debugging (www.eclipse.org/dsdp/dd) in June 2005. The project’s community includes development-tool vendors, operating-system vendors, and semiconductor manufacturers. The project’s mission is to build enhanced debug models, application programming interfaces (APIs), and views that augment the Eclipse Platform for device software development. To date, the Device Debugging project has focused on evolving the existing Eclipse Debug Model into a much more generic and customizable set of interfaces. These interfaces are designed to meet the broad and often diverse requirements of multiple commercial debug solutions.

To accomplish this customization, the project has expanded the Eclipse Platform Debug Interfaces in three key areas:
  • A flexible debug element hierarchy with customized debugger actions
  • Debugger view content and update policies driven entirely by the debug engine
  • Fully asynchronous interaction between the user interface (UI) and debug engine
First, the flexible debug element hierarchy provides the ability for the debug engine to move beyond the process-thread-stack hierarchy that one typically finds in host software development. In device software development--and especially multi-core--the combination of processor types, operating systems, and debug connections defines the hierarchy that’s presented to the developer. For example, consider an arbitrary target board containing a variety of processors and other devices (see Figure 1).

This target has a field-programmable gate array (FPGA) and system-on-a-chip (SoC) device containing two processor cores and a DSP. In a fully integrated development tool that utilizes the flexible debug hierarchy, a developer might see a debug layout as shown in Figure 2. In this example, it’s likely that multiple debug engines are used—perhaps one for the processors, one for the DSP, and one for the FPGA. Each debug engine presents the debuggable contexts that make sense for the connection type and device. With this customization comes the ability to change the meaning of debugger actions at different nodes in the hierarchy. For example, pressing the Go button when selecting a process will restart the process and all of its threads. Pressing Go or Step on a specific thread can allow the debug engine to act only on that thread while leaving the other threads in their current running or stopped states.

Secondly, the new Eclipse Debug Model Interfaces provide the ability for the debug engine to completely customize the content and update policies of the various debugger views in Eclipse (e.g., memory, registers, locals, and expressions). This ability is important because different debug engines provide different levels of visibility into processor resources and control over those resources. For example, a debug engine that provides a JTAG-based connection to a target processor will often allow much greater visibility into processor hardware registers. When connected to that debug engine, the developer therefore expects the Register view to reflect this detail (e.g., bit field descriptions and control, system registers, and coprocessor registers). Because memory and register access on an embedded target can be a timely operation, the new debug-model interfaces allow the debug engine to control how and when view content is updated.

Finally, the new interfaces make all interactions between the GUI and the debug engine asynchronous to improve the debugger’s performance and responsiveness. The benefits of this change are best described by example. Imagine that the application being debugged contains several large data structures that are examined by the developer as he or she steps through code. The developer puts these data structures in the Expression view and also monitors some key processor registers.

As the developer steps through code, all of the data must be updated in each of the views--potentially slowing down the overall stepping speed enough to be an annoyance. By making the interaction with the debug engine asynchronous and allowing the engine to control view updates, the engine can interrupt view refreshing while the user is stepping quickly through code. When the user pauses, the views can refresh completely.

These changes to the Eclipse platform will be released in Eclipse 3.2 in late June of this year as provisional APIs. This customizable debug framework sets the stage for an enhanced multi-core debug experience in which multiple debug engines interact with multiple targets at the same time and share debugger views and a common user experience in Eclipse. Utilizing this new framework, tool vendors can build debugging solutions that focus on their core competencies--processor architectures, operating systems, and hardware debug connectivity methods--while gaining interoperability between multiple solutions.

While these capabilities provide a good starting point for multi-core support, additional work is required to allow the simultaneous and potentially coordinated usage of multiple debug engines. Currently, running multiple debug engines at the same time with the same development language can result in a confusing workflow (e.g., when the user is looking at a C file with two debuggers running and needs to plant a breakpoint using only one debugger). Work also is required in the presentation of data during a multi-core debug session. Eclipse needs to provide debugger views that can be organized into logical groups focused on a specific debugging context. Developers need the ability to group source, expression, register, and memory views for each core together. They also must be able to easily distinguish between views belonging to one core versus another. Furthermore, launch sequences need to be coordinated between cores so that complex systems can be loaded and started simultaneously. These are some of the next areas to be examined by the Device Debugging project.

To learn more about this project, please visit the web site: www.eclipse.org/dsdp/dd. The Device Debugging project is open source. Anyone is welcome to participate by helping define use cases, contributing code, or building commercial products on top of the technology.



Doug Gaff is the PMC Lead for the Eclipse Device Software Development Platform. Gaff also is an Engineering Manager on Wind River’s Eclipse-based development platform, Wind River Workbench.