A construction robot installs a component and reports that the task is complete.
It recorded the position, the tool movement, the torque and the final image. The operation appears successful.
But later, an engineer discovers that the robot used an outdated model revision. The component was installed in the wrong location.
The robot’s experience was real. Its conclusion was wrong.
Now consider what happens if that operation enters a training dataset as a successful example.
The mistake may no longer belong to one robot on one project. It could influence how other machines perform similar work.
As Physical AI moves into factories, construction sites, utilities and infrastructure, this creates a question for owners and operators:
Who decides what a machine’s experience is allowed to teach?
The physical learning loop is getting closer
Robots increasingly combine observations from cameras, LiDAR, sensors and machines with digital models and simulation. Their experience can be used to improve future performance.
One recent signal is Generative Bionics’ plan for a 5,000-square-metre Physical AI Data Factory in Italy, targeted to become operational in the fourth quarter of 2027. The company says it intends to connect robot data collection, simulation, model development and deployment. It is a planned facility, not an operating deployment today. news.gbionics.ai
The idea is compelling: machines perform tasks, generate data, learn and return to the field with improved capabilities.
But the loop raises a practical issue. The system needs to know what actually happened before it decides what the experience means.
A robot can complete its programmed movements without completing the intended job. A drone can capture clear imagery of the wrong asset. A sensor can report a plausible value while operating outside its calibration period.
If those distinctions are lost, scale can spread a poor assumption faster than any individual operator could.
Experience, evidence and outcome are different things
Consider a robot assigned to tighten a bolt on an industrial assembly.
The robot records:
- Its position and tool movement.
- The measured torque.
- Video of the operation.
- The component identifier.
- A “task completed” message.
This is an experience record. It describes what the system observed and did.
To establish whether the bolt was correctly tightened, we need more context:
- Was this the correct component and bolt?
- Was the procedure current?
- Was the tool calibrated?
- Was the measured torque within the required range?
- Did the inspection cover defects that torque alone cannot reveal?
The answers turn observations into evidence for a specific claim. An authorized inspection or acceptance process may then determine whether the installation meets its requirements.
Even that does not automatically answer the final question:
Can this record be used for a particular learning purpose?
A record accepted for one task, robot, tool or operating environment may be unsuitable for another. That decision depends on the intended use, the quality of the data and any restrictions on sharing it.
Failure can be valuable training data
The title needs one important qualification: Physical AI should learn from failures too.
A failed grasp can help a model recognize an unstable object. An interrupted assembly can show when a human intervention was needed. A near miss can reveal a situation that deserves closer testing.
The risk is using a failed or uncertain experience as if it were a verified success.
The REBOOT research project illustrates why the distinction matters. Its precision-assembly dataset includes both expert demonstrations and demonstrations of recovery from failure. The goal is to help robots deal with the kinds of errors that occur during real tasks. It is research evidence, not proof that any particular production robot is safe. nanayawoa.github.io
A practical learning dataset should therefore preserve labels such as:
- Verified success: suitable for a specified learning use.
- Verified failure: potentially useful for failure detection or recovery.
- Uncertain outcome: retained for investigation, with restricted use.
- Invalid or incomplete record: excluded from uses where its limitations make it unreliable.
The question is not simply whether to keep a record. It is what claim the record supports and how it may be used.
Why spatial context changes the answer
In physical environments, the same action can be correct in one place and wrong in another.
A robot may have installed the correct component, used the right tool and achieved the required torque. But if the component belongs to another functional position, the result is still unacceptable.
That is where geospatial information, BIM and digital twins can contribute.
An experience record becomes more useful when it connects the action to:
- The asset and its functional position.
- The location and spatial boundary.
- The applicable design or configuration revision.
- The robot, tool and sensor versions.
- The operating conditions at the time.
- The evidence and acceptance decision.
Imagine that a later inspection finds that several bolts were installed using an incorrectly calibrated tool. If those relationships are preserved, the owner can identify the affected work and investigate whether any related records entered a training dataset.
Without that connection, the organization may have a large collection of robot videos but no reliable way to trace which ones support a particular model or decision.
A digital twin can provide the context for that review. It should not be treated as an infallible judge: its own asset information, revisions and assumptions also need to be checked.
A practical path from action to learning
I see five decisions in a governed physical learning loop:
1. Record the experience. Capture the task, asset, location, configuration, actions, observations and relevant system versions.
2. Validate the evidence. Check identity, source health, calibration, completeness, timestamps and the applicable procedure. Record uncertainty where the evidence is incomplete.
3. Assess the outcome. Compare what happened with the acceptance criteria. For higher-risk work, an independent inspection or authorized human review may be necessary.
4. Decide the permitted use. A verified success may be considered for imitation learning. A verified failure may be appropriate for a failure-detection dataset. An uncertain record may be held for investigation. Privacy, contractual and security restrictions also apply.
5. Test before deployment. A model trained on eligible data still needs evaluation in the intended operating conditions, followed by controlled release, monitoring and a way to suspend or roll back the update.
This last step matters. A well-labelled dataset does not guarantee a safe model. NIST’s AI risk guidance also stresses validation, monitoring and the limits of generalizing beyond the conditions in which a system was developed. AIRC
The decision extends beyond the robotics team
For an asset owner, this is a question of accountability: can the organization trace a physical outcome back to the system, evidence and approval behind it?
For a project leader, it affects rework: can a questionable installation be identified before the same pattern is repeated?
For a robotics provider, it affects deployment: can the provider explain which environments a model has been tested for, and where its limits remain?
For a data owner, it raises another decision: can site recordings be used to train a supplier’s model, shared across customers or retained after a contract ends?
These choices should be made before a learning loop is connected to live operations.
A useful first step is to select one narrow task, define what successful completion means, and trace a small set of records from physical action through inspection and dataset use. The exercise will often reveal missing asset identifiers, inconsistent timestamps or acceptance criteria that were never written down.
The next digital twin question
We have spent years asking whether a digital twin accurately represents an asset.
Physical AI adds another question: does the system preserve enough context to determine what an action can teach?
The strongest learning loop will not be the one that automatically absorbs the most activity. It will be the one that can distinguish an observed event from a supported claim, retain failures honestly and test new behavior before it reaches the physical world.
Because when machines learn from physical work, the quality of the next decision begins with the quality of the last record.
