Use the right methodology to get a better return on your most valuable investment.

The complexity of today’s products requires measurements throughout the complete design and development process. Engineers face the challenge of testing increasingly complicated designs with ever-shrinking timelines and budgets to meet consumer demand for higher-quality products at lower prices. Companies make a significant investment in the data that they collect, from the cost of simulation systems, data acquisition hardware, and automation systems to the associated personnel required to perform and analyze tests. Increasing microprocessor speed and hard drive storage capacity combined with decreasing costs for hardware and software have provoked an explosion of data coming in at a blistering pace. Managing and making good use of this data has become a real challenge.

In today’s fiercely competitive business environment, companies need to rapidly turn raw data into usable information to efficiently drive future product development. Information gained from data acquired throughout each stage of a product’s development cycle can be used to reduce costs, improve quality, or refine processes in order to drive differentiation, optimization, and innovation. The most successful companies in the next decade will be those who make sense of more data than their competitors, faster.

Data is Easier with Context
Engineers often end up selecting a storage technique that most easily meets the needs of the moment, yet data storage can have a sizeable effect on the overall efficiency of the acquisition system and the ultimate scalability of an operation. For example, though custom ASCII files are easily exchangeable, their disk footprint is substantial and any future extension to the file format must be accounted for in custom software. Storage considerations must go beyond the typical concern of file (or database) format alone and include careful planning for storage structure; specifically, the choice of data storage must provide a mechanism for structuring context with raw data.

Data without context is just a series of numbers in a file. By storing raw data with context, it becomes easier to accumulate, locate, and later manipulate. Consider a series of seemingly random integers: 5125558937. At first glance, it is impossible to make sense of this raw information. However, when given context – (512) 555-8937 – the data is much easier to recognize and interpret (a phone number). Measurement data context provides the same benefits, and can describe anything from sensor type, manufacturer, or calibration date for a given measurement channel to revision, designer, or model number for an overall device under test (DUT). In fact, the more context that is stored with raw data, the more the more effectively that data can be traced, searched, and correlated at a future date.

One example of a flexible data storage format that was explicitly designed for measurement data is the industry standard Technical Data Management Streaming (TDMS) file format. The TDMS format is binary (low disk footprint, fast saving and loading), but can be opened in software such as Microsoft Excel. Even more important, TDMS has inherent storage constructs for saving unlimited descriptive information together with measurement data.

The Benefits of a Database…with no IT Intervention?
Data acquisition and storage is only half of the battle. Once data is saved in a well-documented fashion, the challenge becomes management and processing of measurement data. Custom databases initially seem like an attractive data management option due to their searchable nature - scientists and engineers can query data and isolate only the data sets that match a specified set of criteria. However, databases are extremely expensive and complex and can take months or years of planning and programming to implement correctly. Traditional databases also require IT interaction for installation and maintenance, and require modification to the database structure when application requirements change.

File indexing solutions such as the NI DataFinder technology from National Instruments unite the convenience and flexibility of file-based data storage with the searchability of a database. If data is already stored in files with descriptive context, these solutions can extract this meta-information from any file format and build a self-scaling, -adapting, and -configuring database that is abstracted from the user (and their IT department). 

Figure 1. Takata saves thousands of man hours per year with their data management solution.

Reduce the Roadblocks in Data Processing
Imagine immediately identifying only the data channels that were measured using a sensor with a particular serial number, even if those channels are scattered across thousands of data files. Imagine being able to isolate only data from files where a specific threshold level was exceeded, no matter where those files are stored. This type of insight into your data is attainable: an effective data management approach lets you unlock the value in your data investment and analyze data in ways never before possible.

Figure 2. Dedicated processing software simplifies management, analysis and reporting.

As a further return on your data investment, today’s dedicated engineering data processing software will integrate directly with your data management and storage solution. For example, using a template-based reporting technique like that found in DIAdem, you can use variable content placeholders that populate reports automatically using descriptive contextual information from your data file (for instance, to populate the serial number field automatically). If your data processing software is context-aware, it will take measurement units into account when performing analysis calculations on channels with conflicting unit quantities, preventing erroneous calculation results.

Choosing the right data management approach is often an afterthought; but in a world overwhelmed with data, investment in the right data management tools makes all the difference