Contribution guide

In this notebook we will guide you through the basics of contributing to the projective_simulation library.

Parent classes and compulsory modules

To make it easier to develop PS agents and structures, we provide Abstract classes for Agents and Episodic and Compositional Memories (ECM) designed to guide you through the process. These classes include some compulsory modules that must be implemented for your code to qualify as a “true” PS module. Let’s review them so you can start coding your own PS agents!

If you are not yet familiar with PS or how they learn in a Reinforcement Learning environment, we recommend starting with this tutorial.

Agents

In the PS framework, agents must implement two compulsory modules: - deliberate: typically returns an action given an input state
- update: updates the internal structure of your agent

That’s it, the rest is up to you. Let’s look at an example. First, import the Abstract_Agent class from the library:

from projective_simulation.agents.core import Abstract_Agent

And now let’s create our new shinny agent. To illustrate what happens when something goes wrong, let’s “forget” to define the module update:

class my_new_agent(Abstract_Agent):
    
    def __init__(self, num_actions):
        # Takes a random action
        self.num_actions = num_actions

    def deliberate(self):
        return np.random.randint(self.num_actions)

Now let’s call our new class and see what happens:

agent = my_new_agent(3)

TypeError: Cannot instantiate class my_new_agent because it is missing method(s): update.
Please implement all abstract methods (see documentation of class for details).

Ups! As you see, and as expected, the agent is missing the update module. Let’s correct that and double check:

class my_new_BETTER_agent(Abstract_Agent):
    
    def __init__(self, num_actions):
        self.num_actions = num_actions

    def deliberate(self):
        # Takes a random action
        return np.random.randint(self.num_actions)

    def update(self):
            pass

agent = my_new_BETTER_agent(3)

PS: As you can see, the framework does not require the modules to actually perform any operations—only that they are defined. For example, if your research focuses on agents that do not update themselves, you can simply define a dummy update module as shown above.

ECMs

The second main building block of the library is the Episodic and Compositional Memory (ECM). An ECM is typically represented as a graph that stores percepts, encoding observations, internal knowledge, or actions. However, this library does not enforce any specific ECM structure.

The only compulsory module is:

• sample: typically used to generate an action by performing a random walk through the ECM.

Let’s now briefly review what a typical ECM looks like, based on the abstract ECM class (a simplification of projective_simulation.ECMs.core.Two_Layer).

from projective_simulation.ECMs.core import Abstract_ECM

class ECM_2layers(Abstract_ECM):

    def __init__(self,
                 num_actions, # number of actions the agent performs from this ECM
                 num_states # number of different states of the environment
                ):
        # Let's define a "two-layer" h-matrix (see learning PS tutorial for more).
        self.hmatrix = np.random.rand(num_states, num_actions)

    def sample(self, state):
        # Sample greedly the action with the highest h-value
        h_values = self.hmatrix[state]
        action = h_values.argmax()  
        return action

ecm = ECM_2layers(5,5)
ecm.sample(2)

np.int64(4)

And that is all that is fixed by the framwork, from here you are free to create agents and ECMs at will!

Contributing to the projective_simulation library

The nbdev package

This library is based on nbdev. This is a nice tutorial on everything you need to learnd about it.

In the rest of this notebook we will highlight the minimum needed from the library to contribute to the projective_simulation. In particular, we will show you the few magic commands that will allow us to create a python package from notebooks.

Installation

Before starting to write your code, use pipto install the library. To do so, clone the repository and do a local installation of the package. For that, you just need to go to the repo’s folder and run:

pip install -e .

Write new code

Now that you have the library installed, let’s write some new code using Jupyter Notebooks. For the moment, let’s do something simple and talk later about how to properly structure the library. Let’s consider that we want to create, inside the library projective_simulation, a new module utils in which we will gather few useful functions.

To tell nbdev that we want this notebook to do the previous, we use the default_exp magic command in the following way:

Important: any nbdev magic command must be preceded by #|

This will ensure the objects we create within this notebook will be exported to a .py file called contributing_guide.py within the package folder. You can find the latter in the parent repo folder with the name projective_simulation (the chosen name for the library).

Important: never write in those .py, as their content is automatically generated from the notebooks and any change there will be overwritten :) .

Now let’s write some proper code! We will create a function that we want to be contained in the module utils. To do so, we use the command #| export to tell nbdev that we want the content of this to go to the current module:

---

source

random_func

 random_func (k)

We need to actively export the current state of the notebook. To do so, you can either run nbdev_export in your terminal (inside the repo folder) or run the following cell:

import nbdev; nbdev.nbdev_export()

> **Important 1:** `nbdev_export` exports the current state of the notebooks in you library. This means the last saved version! While notebooks usually autosave, it is good practice to save the notebook before running the command.

> **Important 2:** because I don't want this command to go to the module, I don't put the `#| export` command in the cell.

And that’s it! If you are curious, you can go to the repo folder, then projective_simulation/contributing_guide.py and see that the content of the file is exactly the one in the cell above.

Importing generated functions

Because the package has been installed through pip, we can now import the function we just created. Let’s see how. First, restart the notebook’s kernel. Now, you can import the random_func function as:

from projective_simulation.contributing_guide import random_func

random_func(2)

0.31480691257004256

Contributing to the repo

Notebooks have a lot of useless metadata in them (i.e. the count of the cell execution, the environment you are using,…). This would make impossible contribute to a shared repo, as for instance different contributors may be using different environment names, which would end up in a conflict. To avoid this, nbdev has created hooks that deal with this problem (check their documentation if you want to know details). For now, you just need to do the following:

• After installing nbdev (should have been done automatically when installing projective_simulation as I put it as a requirement), run the following in your terminal, inside the repo’s folder:

nbdev_install_hooks

• MOST IMPORTANTLY, before pushing any changes to the origin, be sure to have done two things:
  1. Export the changes you did in the notebook (see above)
  2. Clean the metadata of your notebooks using the command nbdev_clean