Introduction
Manufacturing plants generates immense amount of data through PLCs, SCADA systems, Historians, MES platforms, Sensors, and CMMS or other systems. This generated data helps engineers and floor managers to assess machines, track their productions, and understand what is happening on the floor.
But when the manufacturers try to use these data directly for AI, it creates difficulties, because it is just raw collected data, which alone is not enough to build reliable manufacturing AI insights.
Let us consider an example, if the motor temperature showing as MTR_01_TEMP in one system, MotorTemp_A in second, and TIC101.PV in a third. As we see, the readings utilized for various systems in different units, tags, etc.
So, without having insights on each value it creates difficulties for shop floor person to identify on machine conditions based on timestamps, and it also not making any sense to data which cannot operate for any AI models.
This is where the data preparation becomes important. Before deploying the same raw data into an AI model, manufactures need to collate all the sources on single platform, also make the data consistent and useful based on application, quality checked, and connect with the right operational context.
Here in this blog, we will see the six steps to help manufacturers to identify and filter the right data for AI ready inputs.
But before understanding this process, first we need to go deep with understanding of what kind of challenges factories facing while preparing raw data for AI.
Why Is Raw Manufacturing Data Not Ready for AI?
The challenge is initiating from how the data is being recorded and how the information is being shared via various machines.
We know, PLC controls the machines, SCADA useful for machine operations and data insights, Historian stores the machine readings with time, and MES tracks the production activities.
Here we see, all the systems are generating their own outcomes based on their internal process, so
what happens when they all come together to feed AI models? What kind of challenges they are facing?
Here are some common issues which we must address:
- Different Formats: Systems may utilize OPC UA, MQTT, industrial protocols, databases, or APIs to create and share information. It needs extra engineering efforts such as data integration, normalization, tag mapping, data modelling, time-series synchronization, edge-to-cloud integration etc. to bring all the data together and make it complete understandable view.
- Inconsistent Meaning: If we see value 85 written in any of the machines without having proper contextual meaning whether it represent temperature, pressure, motor speed, or something else including its unit or measurement type. This creates challenge into incorrect analysis, unreliable predictions, and underrated quality and maintenance decisions.
- Different Timestamps: In real-time cases sensors and production systems measures and records information without any specific timestamps. When the readings and production situations do not line up with same time, engineers start facing issues to identify what is the fault in machine: is it from quality checks, or production delay.
- Missing Operating Context: In factory floor, where number of data is generating in every single minute, where we see a sensor reading or alone/isolated is not enough to explain which product is being manufactured, or what kind of load is handling, or when will be the maintenance required. Without having proper context, it leads factory operations stalls, in downtime, and reducing plant performance and affects the operational efficiency.
Now the question raised from factory operators is:
How generated raw data is useful for AI-ready insights?
To mitigate these factory challenges, and improve plant visibility, production time and to bring our generated information/raw data to AI-ready insights; we need to move step by step approach.
Here the first step is to bring out relevant information from bunch of generated data into common structure and processing pipeline with preserving original meaning.
Step One: Bring Data from Different Factory Systems into One Pipeline
In this step we need to identify from where the data generating and how it collects raw information.
We know that machine readings can be collected via protocols using industrial gateways, whereas historian works for storing historical database.
Production and maintenance records can be collected via APIs, and vision systems utilize to provide images for better quality inspection.
During all these collections, it is important to retain some data like, asset ID, source systems, timestamp, data frequency, and original tags and usernames wherever available. Using these details, engineers can identify where the reding comes from and what goes wrong such as unexpected equipment behavior, missing readings, incorrect values, or manufacturing AI insights that do not align with actual factory conditions.
So, the next decision from engineers is where the data is being processed.
For example, high frequency machine readings can be handled by edge gateways which are close to the equipment, which can response quickly without relying on network traffic. Also, centralized processing may be more suitable when we require multiple machine data or production line data to be analyzed together.
This choice depends on some factors like, response time, network bandwidth, data sensitivity, and computing resource availability. Wherever original data should also be preserved so engineers can review early readings or processes based on requirement changes.
So, simply bringing data in this first step itself does not create data consistent. Different systems may still use different tag names, measurement units, and data formats and structures for same readings.
What we need to do to make it more on same page?
The next step will clear the image about to standardise these differences to measure every quantity clearly and consistently to create valuable meaning in factory.
Step Two: Standardize Tags, Units, and Data Structures
Once we record and collect the data from different systems into single pipeline, the next challenge comes to make sure every reading has its consistent meaning.
Different systems use different tag names, units, structures for same measurement of various datasets, which arises difficulties for engineers and floor operators to compare values directly.
Let us, consider a motor temperature recorded by three systems.
Here these values represent same temperature quantity but in various formats (Celsius, Fahrenheit, and Kelvin).
So, here the challenge begins for engineer/floor operator is to verification of units and tag definitions using the continuously generated source metadata and equipment documentation (technical notes or manuals) before treating them as equivalent.
Once verified, the readings can be treated to common structure.
But the question is, before this tabular structure we discussed all data are in various formats, structures and tags.
Then how can we achieve this?
To achieve this, manufactures need to create a tag registry, need mapping of source tags to standard names, conversion of measurements to verified units, and standardize the data types, timestamps, and machine conditions.
These should also retain the original tags and transform the relevant details so that each value can be traced back with the source.
Some standards like, OPC UA information models and ISA-95 can help to establish consistent descriptions of equipment and manufacturing process. ISO 23247 is also useful when we are working with digital twin manufacturing applications.
So, important thing is standardization, which moves beyond with change in tag names. Back with the example of motor temperature, when an AI model receives the data from different machines and production lines, it must be able to interpret the measurements correctly.
But the consistency is the only part of the preparation process.
During consistency, factory floor may find some torn values, missing, outdated, or unrelated values which creates problems and need to solve to make it AI-ready.
In the next step we will check the data quality and identify which data inputs are useful foe the AI application.
Step Three: Check Data Quality and Select What the Model Needs
Once the factory data has been standardized, the next step is about to check whether the collected readings are reliable and contain suitable information for the AI models.
We see consistency in names and units makes data easier to compare, but they can not guarantee that every reading is accurate or useful.
So, we first need data quality assurance. Engineers need to identify missing, skipped readings, duplicate records, outdated values, out of range readings, and communication gaps.
But unusual readings are not always an error.
For example, a sudden rise in motor temperature may be due to faulty sensors or indicate a genuine equipment problem. But without investigating the proper reason, that leads to an early warning.
Therefore, inaccurate readings should be reviewed for analysis, and missing data should be handled based on signal type and application rather than automatically filled without checking.
Now, the selection of the inputs for AI task.
As factory may collects numbers of readings from single machine, but it won’t help to answer the same question of factory: Will it ready to adopt for AI models?
If we are talking about, predictive maintenance model, which is designed to detect bearing problems that need vibration, motor current, temperature, speed, load, and maintenance history. On other side, product quality prediction requires process settings, material batch details, and inspection results.
Engineers can apply their knowledge of the equipment and statistical analysis to identify the relevant inputs and remove unnecessary information when required.
They also need to check that each input will be available when the model makes prediction. So, for example, one could run a model using data recorded only after the failure or maintenance event, thereby incorporating information not available during live operation, which may lead to misleading results.
These checks are supposed to assure that the data is both trustworthy and relevant, so the model can produce dependable manufacturing AI insights.
Yet some aspects must be considered: even reliable readings can be misleading when their time intervals are not synchronized, or if they fall outside the context of analyzing machine operating conditions.
To understand how different signals relate to each other, manufacturers must focus on aligning their timestamps and connect them with the production activities and machine states in which they were recorded.
Step Four: Align Time-Series Data and Add Operating Context
Once the data quality and right input selection activity done by engineers or floor operators, the next step is to align the readings of various machines and connect them with machine’s operating conditions. This is because the readings of such factory systems are taken at different time intervals.
Let us consider an example, a vibration sensor may record readings in every 10 milliseconds, on other end, temperature sensor records the data once per second. Production readings, like a change in batch would be recorded only if they occur.
So, engineers need to align timestamps, adjust the clock differences, and recollect the accurate readings. Missing readings can be filtered out, and interpolation should only be done for any technical reasons.
Once this is done, these values must relate to the relevant operating conditions such as, machine condition, production cycle, batch number/IDs, alarms, and maintenance history by engineers. This helps engineers to understand the background reality of what type of data was recorded.
For example, a temperature rise during normal condition beyond it’s limits is normal, but if it shoots with large amount from steady production, it indicates the problem. Similarly, if high vibration under a heavy load may be expected but unusual when the motor is running at a lower speed.
Through the alignment of readings and association with operational environments, manufacturers can allow AI models to be able to distinguish between what is normal and any faults.
With the readings matched and operational context set, the next process is to prepare the readings for the AI model to use.
Step Five: Prepare Data for the Intended AI Model
Now useful readings are also aligned and connected with appropriate operating conditions, so the next step is to convert them in a format which AI models can utilize. This step is about what the model needs to predict, detect, or understand.
We know different AI applications require different data representations:
- Predictive Maintenance and Time-Series Models: These models may require numerical arrays, sliding time windows, specific features, and specific tags/labels to identify faults in equipment, monitoring of machine health, or predict the future behavior.
- Industrial Knowledge Models: Industrial equipment, sensors, processes, and operating conditions using structured relationship between these parameters to understand the connection between different parts of the manufacturing system.
- Large Language Model (LLM) and Retrieval-Augmented Generation (RAG) Applications: These AI applications may use properly aligned records, JSON objects, alarm descriptions, maintenance records, and manual for equipment arranged for retrieval and contextual understandable responses.
- Vision and Multimodal Models: This vision-based AI models are used for image and video formats which are linked to timestamps, equipment IDs, inspection labels, and related machine conditions to connect visual information with factory operations.
Based on such mentioned applications, process of data preparation may take the form of feature vectors, numerical tensors, structured records, document bifurcations, or input classification that combines different data types.
However, the change in format alone does not make data ready for AI. For example, A PLC tag convert into JSON will not alone help a model to interpret it correctly without any background context of measurement, unreliable supporting values, or lack of context of required outcome.
So, earlier steps are enough to carry this stage towards the model receives the data, which is consistent, relevant, and properly framed.
But how can manufacturers be sure the prepared data will remain reliable when the model moves into production?
Before deployment, engineers need to verify the complete data preparation pipeline, also need to check the same processing rules which are followed consistently, and ensure every input remains constant as factory conditions change.
Step Six: Validate the Data Pipeline Before Deployment
Now manufacturers have structured data, they need to verify the pipeline to ensure the delivery must be reliable. Since machine conditions and production processes can change, they need to validate data and process which they prepare based on the changes.
So, what engineers need to check before deployment? Let us take a look:
- Data Structure and Consistency: This verifies schemas, data types, units, timestamps, targeted value ranges, and whether each measurement has exact meaning.
- Data Traceability: Manufactures can track where the data came from and how it is being processed. Data alignment and conversion rules help engineers to trace issues back with the main source and reproduce the earlier results.
- Engineering Validation: Here data need to compare and prepare with known machine condition and engineering limits. Which helps to identify values that may look valid based on dataset, but it makes no sense under any real operating conditions.
- Data Separation: In the case of time-based systems, separate training, validation, and test sets according to appropriate time intervals or production cycles where necessary. Using random splitting makes it possible for readings that are common in nature within the same time interval to appear in different sets, creating a possibly inflated estimate of model performance.
- Consistent Processing: How manufactures achieve this? Once they apply same data preparation rules during model training and production outcomes. If they found, units, filtering, feature calculations, or other quantities different, the model start behaving differently based on the condition.
- Ongoing Monitoring: This is the final stage where manufacturers/engineers can track the change in data structures, sensor calibration, machine operating conditions, and data distributions. Which helps them to give attention on equipment behavior changes based on factory environment varies.
Ultimately, validation does not end once model goes live. Engineers know that manufacturing conditions varies on every timestamp, so the data pipeline reliability might get affected during constant data input feeding, so to keep it consistent, reliable and relevant to AI models they must go with final check.
Where they can move beyond simply data preparation for single model run and build a pipeline which supports AI application throughout till end.
Building the Industrial Data Foundation for Manufacturing AI Insights
Here let us look, how the whole six stage process looks like in proper designed flow from data collection, normalization to AI model training and deployment based on factory environment requirements:
The complete view looks like:
Factory data sources → Ingestion and source preservation → Normalization and semantic mapping → Data quality and time alignment → Operating context and feature selection → Model-specific data preparation → AI model training and inference
This approach also helps manufacturers make better use of existing equipment and systems by providing a consistent way to interpret data from different sources, without requiring every machine to produce it in the same format.
How MosChip Can Help Build an AI-Ready Industrial Data Foundation
To build an AI-ready data pipeline, the foundation takes more than just model development. It requires factory equipment connectivity, data integration from different sources, edge process of information where needed, and prepare reliable inputs for AI applications.
MosChip’s Industrial IoT services brings machine monitoring, industrial connectivity, and OT-IT integration to support digital manufacturing and connected factory operations. Along with this, we have capabilities in AI engineering services which can cover AI model development, model fine-tuning, and porting of models to different hardware platforms.
On top of this, we have an IoT solution accelerator suite: DigitalSky GenAIoT suite, that includes 100+ foundational models and 50+ Edge AI models that can be fine-tuned based on the use cases.
We work together to combine industrial connectivity, edge computing, data integration, and AI engineering to assist manufacturers in creating data pipelines that will process data from the factory floor so that it is ready for use with specific AI systems.
Wrap Up
Ultimately, manufacturing AI is not only depending upon quantity of data is being produced by any factory floor, but it depends on whether that data is consistent, reliable, connected via appropriate operating context, and it must be prepared for intended AI models.
After addressing such gaps in timestamps, tags, data quality, and production context before model development and deployment, manufacturers can transform disconnected industry operations via useful AI context.
Treating this as a natural engineering process helps manufacturers to deliver reliable, trustworthy manufacturing AI insights despite faces changes from machines, processes, or any such production requirement varies.
To know more about MosChip’s capabilities, drop us a line, and our team will get back to you.