AI Glossary / Reinforcement learning and robotics / Advanced
Relational reinforcement learning
Definition
Relational reinforcement learning combines reinforcement learning with representations of objects and the relationships between them. Instead of treating every named situation as unrelated, it can learn policies or value estimates that depend on reusable relational patterns. Classical approaches use relational learning or inductive logic programming; related neural approaches learn to reason over entities and their interactions.
What it means
Many decision problems involve structured worlds: a job requires a machine, a parcel belongs to an order, or one object blocks another. A relational representation records those connections explicitly or through a suitable learned structure.
The potential benefit is generalization. A rule about a compatible free machine can apply to several machines without learning an entirely separate rule for each identifier. However, transfer depends on the relationships and relevant properties being represented correctly. Renaming an object is different from changing its capabilities.
How it works
The designer or perception system identifies entities, their attributes, and relevant relations. Actions may also be parameterized by entities, such as assigning job J to machine M. The learner receives rewards from experience and learns how relational descriptions predict useful actions or future return.
A classical implementation can learn logical conditions or relational decision structures for action values. A neural implementation may use attention or graph-style computation to combine information across entities. These are different mechanisms; the common concern is representing and exploiting relationships in a sequential decision problem.
Variables allow a pattern to refer to objects by role rather than by a fixed name. For example, a candidate condition might require that a machine is free and compatible with a job. An action satisfying that condition still needs a learned or otherwise established value; relational syntax alone does not determine the best decision.
Evaluation should include different object identities, combinations, and problem sizes. A policy that works after renaming entities may still fail when there are more entities or new kinds of dependencies. Relational generalization is an objective to test, not a universal consequence of choosing a structured representation.
Three practical examples
- Scheduling research: represent compatibility and precedence relations among jobs and resources.
- Object manipulation: describe which objects support, obstruct, or are reachable from other objects.
- Structured game environments: learn decisions involving units, locations, and dependencies rather than only fixed object identifiers.
Think of it this way
Think of learning the instruction assign a ready job to a compatible free machine instead of memorizing job 17 goes to machine 4. The first form can survive changes in names, but it still needs to account for processing time, capacity, and other facts that make one assignment better than another.
Worked example
Consider a fictional one-decision scheduling task with jobs J1 and J2 and machines M1 and M2. Both machines are free. The recorded compatibility relations are compatible(J1,M1), compatible(J1,M2), and compatible(J2,M2). There is no compatibility relation between J2 and M1, and the task explicitly treats missing compatibility facts as false.
Each feasible assignment ends the episode. Completing J1 yields 12 points and completing J2 yields 9. A compatible assignment to M1 succeeds with probability 0.80 and costs 2 points. A compatible assignment to M2 succeeds with probability 0.95 and costs 4. Failure earns no completion points, but the assignment cost is still paid.
The expected rewards are 0.80 times 12 - 2 = 7.6 for J1 to M1; 0.95 times 12 - 4 = 7.4 for J1 to M2; and 0.95 times 9 - 4 = 4.55 for J2 to M2. The best feasible assignment under these assumptions is J1 to M1. These values can serve as targets for checking a learned policy; supplying them analytically is not itself reinforcement learning.
Now rename J1 as Red and M1 as West while preserving all attributes and relationships. A representation and policy that depend on the relevant roles and properties should preserve the corresponding ranking. Memorizing the literal strings J1 and M1 would not provide that generalization.
Next change West's success probability to 0.50. Its expected reward for Red becomes 0.50 times 12 - 2 = 4. The other machine's 7.4 now wins. A rule that always chooses the formerly preferred machine would be wrong because the underlying properties changed.
Finally, adding more jobs would create a different decision problem if assignments affected later capacity. The one-step calculations would then be insufficient; future consequences would need to enter the values. Relational representation does not remove the sequential nature of reinforcement learning.
Illustrative case study
Fictional case: RowanPlanning tests a relational scheduling learner. It performs well on familiar machine names, but its accuracy falls when the same simulated facility is relabeled. The team discovers that an input field encourages the model to rely on identifiers rather than compatibility and cost relationships.
It revises the representation and separates three tests: renamed instances, new combinations of known properties, and larger scheduling problems. This invented case illustrates how to investigate generalization; it makes no claim that a particular architecture will pass all three tests.
Common misconceptions and mistakes
Common mistakes include confusing relational reinforcement learning with reinforcement learning stored in a relational database, assuming every relational policy is human-readable, and treating graph structure as a guarantee of transfer. Another mistake is omitting important properties because objects appear similar. Different capabilities, costs, or hidden constraints can make superficially identical relationships lead to different optimal actions.
Business use
Businesses can explore relational methods when decisions repeatedly involve varying collections of entities and reusable relationships. A practical assessment should compare representation costs, data requirements, inference time, and performance on genuinely new configurations. Building reliable entity and relationship information may be as important as the learning algorithm, particularly when perception or business records are incomplete.
People also ask
Does relational mean SQL-based?
No. It refers to relationships in the problem representation.
Does it always use logic rules?
Classical methods often do; neural relational approaches also exist.
Can it handle more objects than seen in training?
Sometimes, but that capability requires explicit testing and depends on the method and task.
Check your understanding
Which assignment has expected reward 7.6?
Answer: J1 to M1 under the original assumptions.
What should renaming alone preserve?
Answer: The corresponding decision ranking when all relevant structure is unchanged.
Why does changing success probability matter?
Answer: It changes the expected outcome, not just the object's label.
Related terms
- Reinforcement learning
- Inductive logic programming
- Graph neural network
- Attention
- Generalization
- relational learning
- object representation
- relational policy
- action-value function
Learn next
Next, represent a small decision task using entities, attributes, and relations. Test whether a simple learned policy survives consistent renaming before evaluating new configurations and larger numbers of objects.
More in Reinforcement learning and robotics
- Reinforcement - Reinforcement is evaluative feedback that influences how an agent learns from its behavior.
- Reinforcement Learning from Human Feedback - Reinforcement Learning from Human Feedback, commonly abbreviated RLHF, is a family of methods that uses human evaluations to guide reinforcement learning.
- Reinforcement learning with verifiable rewards
- Replacing eligibility trace - A replacing eligibility trace is a short-term credit-assignment record that is reset to a fixed active value, usually one, whenever its state or feature is...
- Residual-gradient reinforcement learning - Residual-gradient reinforcement learning updates a value approximator by differentiating a squared Bellman-style residual through both the current...
- Retrace(lambda) - Retrace(lambda) is a multi-step off-policy reinforcement-learning method that combines target-policy temporal-difference errors using truncated...
Sources
Saso Dzeroski, Luc De Raedt, and Kurt Driessens, Relational Reinforcement Learning: https://dtai-static.cs.kuleuven.be/publications/files/34461.pdf Vinicius Zambaldi and colleagues, Relational Deep Reinforcement Learning: https://arxiv.org/abs/1806.01830
Previous: Relational Model/Tasmania Next: Relational Synthesis Algorithm