Learning KnowRob

date

slug

knowrob

author

status

Public

Learning KnowRob

This post is a summary of my notes/takes while learning KnowRob.

What is KnowRob?

A brief description of my understanding of what KnowRob is:

KnowRob is a knowledge processing system consisting of a knowledge base (KB), knowledge management interfaces, and reasoning mechanisms.

In practical terms, the knowledge base is represented as an OWL ontology, which internally gets converted to a triple format and stored in a MongoDB database. Knowledge management is performed with custom KnowRob Prolog predicates, which can be queried through a KnowRob command line interface or through code in ROS nodes. The reasoning process is carried out with user-defined Prolog rules when the KB is queried, the inferred facts are virtualized, that is, they are available only at the time of the query and are not stored in the KB.

To fully understand what KnowRob is, check the KnowRob papers, its website, and KnowRob’s repo README.

If you are going for the papers, I recommend checking these 2 first:

KnowRob: A knowledge processing infrastructure for cognition-enabled robots

Know Rob 2.0 — A 2nd Generation Knowledge Processing Framework for Cognition-Enabled Robotic Agents

Installing KnowRob

At the time of this writing (10/08/2023), KnowRob's master branch is intended to be used with Ubuntu 20.04 and ROS noetic. Since for my work I am already using ROS 2, I have Ubuntu 22.04 in my work laptop. Thus, to use KnowRob, I need to run it with docker.

There is a dockerfile and a docker-compose file in the KnowRob repository, however it is outdated. So, I created the repository knowrob_docker to store an updated version of it, and I added a CI to rebuild the image every week and update the docker image registry.

You can either use the docker image in the registry ghcr.io/rezenders/knowrob_docker:main or you can build the image yourself. If you choose to build it, just follow the instructions in the README, and replace the image name in the docker-compose file.

Running KnowRob

To run KnowRob, you need to start roscore, the KnowRob node, any additional nodes that should interact with KnowRob, and mongodb.

Docker

If you are using docker with the dockerfile I provide, you can start roscore + KnowRob + mongodb by starting the docker_compose file:

docker-compose up

Note: Don’t forget to install docker and docker-compose

Then you need to run any additional nodes within a docker container connected to the knowrob_network. An example:

docker run -it --rm --net knowrob_network --name rosprolog_cli --env ROS_MASTER_URI=http://knowrob:11311 ghcr.io/rezenders/knowrob_docker:main rosrun rosprolog rosprolog_commandline.py

Note: This docker image only contains KnowRob, if you want to use it with your own custom package, you need to create a new image from this one. Check these examples of a custom dockerfile and docker-compose.yml

Bare metal (without docker)

MongoDB:

sudo systemctl start mongod.service

roscore:

roscore

Knowrob:

roslaunch knowrob knowrob.launch

Additional nodes, example:

rosrun rosprolog rosprolog_commandline.py

Note: In the rest of the post I am only going to provide instructions on how to run with docker, just adapt the commands if you are using it bare metal

Example of how to use KnowRob

To exemplify how KnowRob can be used, I will use the content of the knowrob_intro repository.

A snippet of one of the OWL ontologies contained in the repository can be seen below:

Snippet of the ontology from the knowrob_intro repo

So given this ontology, how can we use KnowRob to discover:

Which types of food are perishable?

Which types of food are not perishable?

What types of Robot it contains?

Is Honey non perishable?

What are the instances of Food?

What is the weight of a certain instance of Milk?

These can be discovered by querying the knowledge base, which in KnowRob is done with custom Prolog queries. Let’s check how we can do this below.

Note: In the snippet above everything is an Entity, no Individuals are shown

Start KnowRob with the knowrob_intro package

To begin with this example, we need to start KnowRob, mongoDB, and the rosprolog command line interface using the docker image from the knowrob_intro repository.

First, clone the knowrob_intro repo:

cd ~/
git clone https://github.com/kas-lab/knowrob_intro.git

Then, start KnowRob + mongoDB:

cd ~/knowrob_intro
docker-compose up

In another terminal, start the prolog command line:

docker run -it --rm --net knowrob_network --env ROS_MASTER_URI=http://knowrob:11311 ghcr.io/kas-lab/knowrob_intro:main rosrun rosprolog rosprolog_commandline.py

Query using rosprolog command line interface

Now, we can load an ontology and start interacting with it.

In the prolog command line terminal, run:

load_owl('package://knowrob_intro/owl/krr_exercise.owl', [namespace(pap, 'http://www.airlab.org/tiago/pick-and-place#')])

This command loads the krr_exercise.owl ontology into knowrob with the namespace pap. If everything worked, this command should have returned true.

Now we can query the KB to find the answers for the questions above.

Question 1: Which types of food are perishable?

To find an answer for this question, we can run the following query in the prolog terminal:

subclass_of(P, pap:'Perishable')

You should get the following result:

P: http://www.airlab.org/tiago/pick-and-place#Cheese ;

P: http://www.airlab.org/tiago/pick-and-place#Milk ;

P: http://www.airlab.org/tiago/pick-and-place#Yogurt.

If we compare this result with the figure showing a snippet of the ontology, we can see that the answer is correct.

Note: You need to press ENTER to get the next result, in this case you will need to press it 3 more times.

Question 2: Which types of food are not perishable?

?- subclass_of(P, pap:'NonPerishable').
P: http://www.airlab.org/tiago/pick-and-place#Hagelslag ;

P: http://www.airlab.org/tiago/pick-and-place#Honey ;

P: http://www.airlab.org/tiago/pick-and-place#Rice.

Question 3: What types of Robot it contains?

?- subclass_of(P, pap:'Robot').
P: http://www.airlab.org/tiago/pick-and-place#Tiago.

Question 4: Is Honey non perishable?

?- subclass_of(pap:'Honey', pap:'NonPerishable').
true.

Question 5: What are the instances of Food?

?- instance_of(X, pap:'Food').
X: http://www.airlab.org/tiago/pick-and-place#hagelslag1 ;

X: http://www.airlab.org/tiago/pick-and-place#honey1 ;

X: http://www.airlab.org/tiago/pick-and-place#milk1 ;

X: http://www.airlab.org/tiago/pick-and-place#rice1 ;

X: http://www.airlab.org/tiago/pick-and-place#yogurt1._

Question 6: What is the weight of a certain instance of Milk?

From the result of question 5, we know that there is an individual of Milk named milk1. Let’s find out what is the weight of this individual.

To get milk1's weight, we can query its hasMassAttribute property

?- triple(pap:'milk1', soma:'hasMassAttribute', X).
X: http://www.airlab.org/tiago/pick-and-place#weight_value_milk.

With this query, we discovered that there is an individual weight_value_milk that holds milk1's weight. We can query its hasMassValue to finally find out its weight

?- triple(pap:'weight_value_milk', soma:'hasMassValue', X).
X: 0.7.

Add new knowledge to KB

To add knowledge you can use the kb_project predicate.

For example, to add milk2 as an instance of Food:

?- kb_project(instance_of(pap:'milk2', pap:'Food')).
true.
?- instance_of(X, pap:'Food')
X: http://www.airlab.org/tiago/pick-and-place#hagelslag1 ;

X: http://www.airlab.org/tiago/pick-and-place#honey1 ;

X: http://www.airlab.org/tiago/pick-and-place#milk1 ;

X: http://www.airlab.org/tiago/pick-and-place#rice1 ;

X: http://www.airlab.org/tiago/pick-and-place#yogurt1 ;

X: http://www.airlab.org/tiago/pick-and-place#milk2.

Add mass attribute to milk2:

?- kb_project(instance_of(pap:'mass_milk2', soma:'MassAttribute')).
true.
?- kb_project(triple(pap:'mass_milk2',soma:'hasMassValue', 0.7)).
true.
?- kb_project(triple(pap:'milk2', soma:'hasMassAttribute', pap:'mass_milk2')).
true.

You can query the KB to check if everything went well:

?- instance_of(M, soma:'MassAttribute')
M: http://www.airlab.org/tiago/pick-and-place#milk_weight_full ;

M: http://www.airlab.org/tiago/pick-and-place#rice_weight_half ;

M: http://www.airlab.org/tiago/pick-and-place#weight_value_hagelslag ;

M: http://www.airlab.org/tiago/pick-and-place#weight_value_honey ;

M: http://www.airlab.org/tiago/pick-and-place#weight_value_milk ;

M: http://www.airlab.org/tiago/pick-and-place#weight_value_yogurt ;

M: http://www.airlab.org/tiago/pick-and-place#mass_milk2.

?- triple(pap:'mass_milk2', soma:'hasMassValue', X).
X: 0.7.

?- triple(pap:'milk2', soma:'hasMassAttribute', X).
X: http://www.airlab.org/tiago/pick-and-place#mass_milk2.

Remove knowledge from KB

To remove knowledge you can use the kb_unproject predicate

To remove milk2 and its mass attribute:

?- kb_unproject(triple(pap:'milk2', soma:'hasMassAttribute', pap:'mass_milk2'))
true.
?- kb_unproject(triple(pap:'mass_milk2',soma:'hasMassValue', 0.7)).
true.
?- kb_unproject(triple(pap:'mass_milk2', rdf:'type', soma:'MassAttribute')).
true.
?- kb_unproject(triple(pap:'milk2', rdf:'type', pap:'Food')).
true.

You can query the KB to check if everything was deleted correctly:

?- triple(pap:'milk2', soma:'hasMassAttribute', X).
false.

?- triple(pap:'mass_milk2', soma:'hasMassValue', X).
false.

?- instance_of(M, soma:'MassAttribute')
M: http://www.airlab.org/tiago/pick-and-place#milk_weight_full ;

M: http://www.airlab.org/tiago/pick-and-place#rice_weight_half ;

M: http://www.airlab.org/tiago/pick-and-place#weight_value_hagelslag ;

M: http://www.airlab.org/tiago/pick-and-place#weight_value_honey ;

M: http://www.airlab.org/tiago/pick-and-place#weight_value_milk ;

M: http://www.airlab.org/tiago/pick-and-place#weight_value_yogurt.

?- instance_of(X, pap:'Food')
X: http://www.airlab.org/tiago/pick-and-place#hagelslag1 ;

X: http://www.airlab.org/tiago/pick-and-place#honey1 ;

X: http://www.airlab.org/tiago/pick-and-place#milk1 ;

X: http://www.airlab.org/tiago/pick-and-place#rice1 ;

X: http://www.airlab.org/tiago/pick-and-place#yogurt1.

Note:

To add a new instance of a class we used kb_project/1 + instance_of/2, e.g., kb_project(instance_of(M, soma:'MassAttribute')).

However, kb_unproject/1 doesn’t work with instance_of/2, thus, we need to use it in combination with triple(instance, rdf:'type', class), e.g., kb_unproject(triple(pap:'milk2', rdf:'type', pap:'Food')).

Extra predicates for querying

In the example above, we showed how some predicates can be used to query the KB, such as subclass_of, instance_of, and triple. However, there are many more predicates available. I have to confess that I didn’t find out if there is a page containing ALL predicates available, but you can check some other predicates in the pages:

https://knowrob.github.io/knowrob/master/lang/index.html

https://knowrob.github.io/knowrob/master/model/index.html

Query via code in ROS nodes

To query KnowRob with code, we need to create a ROS node and query from within this node. The queries have the same format as the ones used in the example above, the only difference is that they are not called with the rosprolog command line but with python code.

Check the example_node.py in the knowrob_intro package. Note that this node uses the PrologQuery class to make querying easier.

To run this example, first start knowrob_intro as we did before:

cd ~/knowrob_intro
docker-compose up

And in another terminal, instead of rosprolog, run:

docker run -it --rm --net knowrob_network --env ROS_MASTER_URI=http://knowrob:11311 knowrob_intro rosrun knowrob_intro example_node.py

After running the second docker container with the example_node, a lot of stuff should have been printed. Let's dive into the queries that generated this output.

What are the instances of Milk?

The following query returns all instances of the Milk class

# (Find) all instances of milk
query = pq.prolog_query("instance_of(X, pap:'Milk').")
print("Instances of Milk:")
pq.get_all_solutions(query)

The correspondent result should be something like this:

Instances of Milk:
X : milk1

It means that there is only one instance of Milk, and that this instance is milk1

What are the instances of Food?

The following query returns all instances of the Food class

# (Find) all instances of Food
query = pq.prolog_query("instance_of(X, pap:'Food').")
print("Query result:")
print("Instances of Food:")
food_instances = pq.get_all_solutions(query)
print("First food instance found {}".format(food_instances[0]))

The correspondent result should be:

Instances of Food:
X : hagelslag1
X : honey1
X : milk1
X : rice1
X : yogurt1

It means that there are 5 instances of Food

What is the weight of a certain instance of Milk?

The following query returns the mass value of milk1

query = pq.prolog_query("holds(pap:'milk1', soma:hasMassAttribute, X)")
print("Mass attribute of milk1:")
pq.get_all_solutions(query)
query = pq.prolog_query("holds(pap:weight_value_milk, soma:hasMassValue, X)")
print("Value of mass attribute of milk1:")
pq.get_all_solutions(query)

The correspondent result should be:

Mass attribute of milk1:
X : weight_value_milk
Value of mass attribute of milk1:
X : 0.7

This means that milk1 weights 0.7

Note: Differently from the example with rosprolog, we used the hold predicate instead of triple, but triple should also work in this case.

Query with reasoning and custom predicates

So far, we covered how to query the knowledge base using general KnowRob predicates, such as triple or instance_of. With these queries, you can retrieve knowledge that is explicitly modeled/included in the knowledge base. However, one of the most interesting aspects of KnowRob is its ability to infer new facts that are not explicitly modeled through reasoning. This is achieved with custom Prolog rules.

Let’s use the knowrob_intro package again as an example, check the owl_kb_prolog_rules.pl Prolog module.

In the first part of the file, the custom predicates that this module implements are declared. Thus, when Knowrob loads this module, these predicates will be available for querying, e.g., is_perishable('milk1'). Note that /1 indicates the arity of the predicate, i.e., the number of arguments it takes.

% module definition and public (externally visible) predicates
:- module(owl_kb_prolog_rules,
    [
        is_perishable/1,
        is_nonperishable/1,
        location/2,
        transfer/3
    ]).

After declaring the available predicates, it is necessary to define what they do. In the following example, is_perishable(X) is equal to instance_of(X, pap:'Perishable'). , i.e., it is basically used to shorten the syntax. Similar for is_nonperishable(X) and location(X, L).

is_perishable(X) :- instance_of(X, pap:'Perishable').
is_nonperishable(X) :- instance_of(X, pap:'NonPerishable').
location(X,L) :- triple(X, dul:'hasLocation', L).

The following example is a bit more involved, transfer finds a refrigerated table (L2) where the perishable items (X) that are in a non refrigerated table (L1) can be placed, and the opposite for non-perishable items.

% returns if the transfer operation is correct accordng to the rule that
% perishable items should be in a refrigerated table
transfer(X, L1, L2) :- is_perishable(X),
                        location(X,L1),
                        instance_of(L1, pap:'NonRefrigeratedTable'),
                        instance_of(L2, pap:'RefrigeratedTable').

transfer(X, L1, L2) :- is_nonperishable(X),
                    location(X,L1),
                    instance_of(L1, pap:'RefrigeratedTable'),
                    instance_of(L2, pap:'NonRefrigeratedTable').

Let’s run this example. Edit the docker-compose.yml file by replacing command: roslaunch knowrob knowrob.launch with command: roslaunch knowrob_intro example_owl_kb_and_prolog_rules.launch. Then, run docker compose:

cd ~/knowrob_intro
docker-compose up

The output should be something like:

knowrob_container | D : table_1
knowrob_container | O : table
knowrob_container | X : milk1
knowrob_container | D : table_1
knowrob_container | O : table
knowrob_container | X : yogurt1
knowrob_container | D : table
knowrob_container | O : table_1
knowrob_container | X : hagelslag1
knowrob_container | D : table_0
knowrob_container | O : table_1
knowrob_container | X : hagelslag1
knowrob_container | D : table
knowrob_container | O : table_1
knowrob_container | X : honey1
knowrob_container | D : table_0
knowrob_container | O : table_1
knowrob_container | X : honey1

This means that milk1 can be moved from table to table1. Note that this knowledge about transfers was not explicitly included in the knowledge base, it was inferred using the transfer rule.

You can find more examples of custom predicates in the sc_planning.pl module.

Custom KnowRob package

You can find information on how to create your own KnowRob package in KnowRob’s official documentation. But let’s recap here anyway, using the knowrob_intro package again as an example. To create your custom package you need to:

Create a new ROS package for your project

Include an __init__.pl file in the src folder, registering the package and including all custom prolog modules you define

Include your custom prolog modules in the src folder, with all the custom predicates you want to use

Add launch files pointing rosprolog to your package

If you want to run it with docker, add a dockerfile and docker-compose.yml with your stuff

That is it!

References and relevant links

Extra example: metacontrol implementation with knowrob

KnowRob Website

knowrob_intro package

Papers

To achieve its task, the AUV is capable of performing