Update documentation

README.md

@@ -1,6 +1,6 @@
 # Website
 
-This website is built using [Docusaurus 2](https://docusaurus.io/), a modern static website generator.
+This website is built using [Docusaurus](https://docusaurus.io/), a modern static website generator.
 
 ### Installation
 

docs/api/agents/_category_.json

@@ -0,0 +1,7 @@
+{
+  "label": "Agents",
+  "position": 1,
+  "link": {
+    "type": "generated-index"
+  }
+}

docs/api/agents/graph-program-interpreter.md

@@ -0,0 +1,62 @@
+# Graph Program Interpreter
+
+The Graph Program Interpreter serves as the primary agent within the HybridAGI framework. Similar to how a Python interpreter processes a program line by line, this agent interprets a graph program node by node. This approach enables the agent to demonstrate explainable behavior, as it executes only the actions and decisions specified by the graph. The deterministic nature of this behavior is vital for the development of explainable systems and allows developers to improve and correct the agent's behavior without the need for fine-tuning.
+
+In contrast to React agents, which depend heavily on the training of Language Language Models (LLMs) to manage new tasks, HybridAGI has the ability to direct the agent to act beyond the limits of its data distribution. This capability enables HybridAGI to handle previously unencountered tasks with ease and control.
+
+By not permitting the agent system to decide which tool to use at each step, HybridAGI can accommodate an infinite number of tools. This is a significant advantage over React-based agent architectures or agents that rely on function calling. Furthermore, this mechanism allows for the use of smaller LLMs that have not been specifically trained to use function calling or tools.
+
+## Usage
+
+```python
+from hybridagi import GraphProgramInterpreter
+from hybridagi import ProgramMemory, TraceMemory
+from hybridagi import SentenceTransformerEmbeddings
+from hybridagi import AgentState
+
+# The embeddings needed for the hybrid vector/graph memories
+embeddings = SentenceTransformerEmbeddings(
+    dim = 384,
+    model_name_or_path = "sentence-transformers/all-MiniLM-L6-v2",
+)
+
+# The program memory is mandatory for the agent system
+program_memory = ProgramMemory(
+    index_name = "hybrid_agi",
+    embeddings = embeddings,
+    wipe_on_start = True,
+)
+
+# The trace memory is optional but useful to inspect the behavior of your agent
+trace_memory = TraceMemory(
+    index_name = "hybrid_agi",
+    embeddings = embeddings,
+    wipe_on_start = True,
+)
+
+tools = [
+    # ...
+    # The tools to use
+]
+
+agent_state = AgentState()
+
+interpreter = GraphProgramInterpreter(
+    program_memory = program_memory, # The program memory (mandatory)
+    trace_memory = trace_memory, # The trace memory that records every step inside the graph database
+    agent_state = agent_state, # The state of the agent, created if not provided
+    tools = tools, # The tools to use
+    entrypoint = "main", # The entrypoint of the interpreter
+    num_history = 5, # The number of steps to include in the agent context (default is 5)
+    max_iters = 20, # The maximum iterations of the agent (default is 20)
+    add_final_step = True, # Whether or not to add an extra step for the final answer (default to True)
+    commit_decision_steps = True, # Whether or not to include decision steps in the agent context (default is True)
+    commit_program_flow_steps = True, # Whether or not to include the call and end program steps in the context (default is True)
+    return_final_answer = True, # Whether or not to return the agent's final answer (default to True)
+    return_program_trace = True, # Whether or not to return the agent trace (default to True)
+    return_chat_history = True, # Whether or not to return the chat history (default to True)
+    verbose = True, # Prints the interpreter steps if True
+)
+
+
+```

docs/api/embeddings/_category_.json

@@ -0,0 +1,7 @@
+{
+  "label": "Embeddings",
+  "position": 2,
+  "link": {
+    "type": "generated-index"
+  }
+}

docs/api/embeddings/custom-embeddings.md

@@ -0,0 +1,27 @@
+---
+sidebar_position: 3
+---
+
+# Adding Custom Embeddings
+## Create your own embeddings
+
+When adding a custom embeddings, you will need to create an Object that inherit from `BaseEmbeddings`, here is the interface to follow:
+
+```python
+import abc
+import numpy as np
+from typing import Union, List
+
+class BaseEmbeddings():
+
+    def __init__(self, dim: int):
+        self.dim = dim
+
+    @abc.abstractmethod
+    def embed_text(self, query_or_queries: Union[str, List[str]]) -> np._typing.NDArray:
+        pass
+
+    @abc.abstractmethod
+    def embed_image(self, image_or_images: Union[np._typing.NDArray, List[np._typing.NDArray]]) -> np._typing.NDArray:
+        pass
+```

docs/api/embeddings/fake-embeddings.md

@@ -0,0 +1,22 @@
+---
+sidebar_position: 2
+---
+
+# Fake Embeddings
+## Embeddings for testing purposes
+
+Fake Embeddings are an indispensable resource for ensuring the smooth and efficient testing of your applications. They generate a random vector of the specified dimension, making them perfect for simulating real-world embeddings during the testing phase. By incorporating Fake Embeddings into your testing pipelines, you can effectively evaluate the functionality of your applications without the need for actual embeddings.
+
+## Usage
+
+```python
+import unittest
+from hybridagi import FakeEmbeddings
+
+def test_embeddings():
+    emb = FakeEmbeddings(dim=250)
+    vector = emb.embed_text("hello world")
+    assert len(vector) == 250
+```
+
+By integrating Fake Embeddings into your testing workflows, you can streamline your testing processes and ensure the robustness and reliability of your applications.

docs/api/embeddings/sentence-embeddings.md

@@ -0,0 +1,23 @@
+---
+sidebar_position: 1
+---
+
+# Sentence Transformer Embeddings
+## Embeddings for text data
+
+Sentence Transformer Embeddings are used for computing the vectors used in similarity retrieval. These embeddings are an essential component for each `HybridStore`, empowering you to fetch text data efficiently.
+
+# Usage
+
+```python
+from hybridagi import SentenceTransformerEmbeddings
+
+embeddings = SentenceTransformerEmbeddings(
+    model_name_or_path = "sentence-transformers/all-MiniLM-L6-v2", # The name of the model to use
+    dim = 384, # The dimension of the embeddings vector
+    max_gpu_devices = 1, # The maximum number of GPU to use (default to 1)
+    batch_size = 256, # The maximum of embeddings to compute in one batch (default to 256)
+    max_seq_length = 256, # The maximum number of input tokens for the embeddings (default to 256) 
+    normalize_embeddings = True, # Whether or not to normalize the embeddings (default to True)
+)
+```

docs/api/hybridstores/_category_.json

@@ -0,0 +1,7 @@
+{
+  "label": "Hybridstores",
+  "position": 3,
+  "link": {
+    "type": "generated-index"
+  }
+}

docs/api/hybridstores/fact-memory.md

@@ -0,0 +1,60 @@
+---
+sidebar_position: 4
+---
+
+# Fact Memory
+## HybridAGI triplet store
+
+The Fact Memory allows the agent system to access a knowledge graph representing entities and their relations, enabling GraphRAGs and other knowledge-intensive applications. By indexing the entities, it enables the agent to search for a specific entity to answer questions or to verify facts.
+
+This data structure is classical in knowledge-based systems like in robotics, biology or medecine. However, there is one drawback: the knowledge graphs are domain-specific, so the structure will vary significantly depending on your application. We provide some knowledge parsers to help you leverage this, but you will likely have to implement your own knowledge parser depending on your application.
+
+<figure>
+  <p align="center">
+    ![HybridAGI fact memory](img/fact_memory.png)
+    <figcaption align="center"><b>Fig.1 - HybridAGI's fact memory indexes each entity, allowing graph augmented generation.</b></figcaption>
+  </p>
+</figure>
+
+## Usage
+
+```python
+from hybridagi import ProgramMemory, FactMemory
+from hybridagi import SentenceTransformerEmbeddings
+from hybridagi.tools import (
+    EntitySearch, # allows the system to search for entities
+)
+
+embeddings = SentenceTransformerEmbeddings(
+    dim = 384,
+    model_name_or_path = "sentence-transformers/all-MiniLM-L6-v2",
+)
+
+program_memory = ProgramMemory(
+    index_name = "hybrid_agi", 
+    embeddings = embeddings,
+)
+
+fact_memory = FactMemory(
+    index_name = "hybrid_agi", # The global index
+    embeddings = embeddings, # The embeddings to use
+    graph_index = "fact_memory", # The hybridstore index (default to fact_memory)
+    hostname = "localhost", # FalkorDB hostname (default to localhost)
+    port = 6379, # FalkorDB port (default to 6379)
+    username = "", # FalkorDB username (empty by default)
+    password = "", # FalkorDB password (empty by default)
+    indexed_label = "Entity", # The label of the indexed nodes (default to Entity)
+    wipe_on_start = False, # Whether or not to wipe the hybridstore at start (default to False)
+)
+
+tools = [
+    EntitySearch(
+        fact_memory = fact_memory
+    ),
+]
+
+interpreter = GraphProgramInterpreter(
+    program_memory = program_memory,
+    tools = tools,
+)
+```

docs/api/hybridstores/file-system.md

@@ -0,0 +1,92 @@
+---
+sidebar_position: 2
+---
+
+# File System
+## HybridAGI document store
+
+The File System is HybridAGI's document store, unlike vector-only databases, it preserves the structural integrity of the documents and folders. This allows the system to maintain the semantic separation between your documents.
+
+When combined with the tools we provide, the agent can navigate inside this document store in a Unix-like fashion, enhancing its ability to efficiently search for and retrieve information.
+
+<figure>
+  <p align="center">
+    ![HybridAGI filesystem](img/filesystem.png)
+    <figcaption align="center"><b>Fig.1 - HybridAGI's long-term memory implements a file system-like structure of folders and files. This architecture enables the AI system to navigate and explore its vector store in a Unix-like fashion.</b></figcaption>
+  </p>
+</figure>
+
+## Usage
+
+```python
+from hybridagi import ProgramMemory, FileSystem
+from hybridagi import GraphProgramInterpreter
+from hybridagi import SentenceTransformerEmbeddings
+from hybridAGI import AgentState
+# The tools to access the filesystem
+from hybridagi.tools import (
+    ReadFile, # Read document chunk by chunk
+    WriteFile, # Write into a file (override if existing)
+    AppendFile, # Append into a file (create one if non-existing)
+    DocumentSearch, # Perform a similarity based search 
+    InternalShell, # Use the Unix-like shell to navigate into the hybridstore and organize it
+    Upload, # Zip and download a folder or file from the hybridstore
+)
+
+embeddings = SentenceTransformerEmbeddings(
+    dim = 384,
+    model_name_or_path = "sentence-transformers/all-MiniLM-L6-v2",
+)
+
+program_memory = ProgramMemory(
+    index_name = "hybrid_agi", 
+    embeddings = embeddings,
+)
+
+agent_state = AgentState()
+
+filesystem = FileSystem(
+    index_name = "hybrid_agi", # The global index
+    embeddings = embeddings, # The embeddings to use
+    graph_index = "filesystem", # The hybridstore index (default to filesystem)
+    hostname = "localhost", # FalkorDB hostname (default to localhost)
+    port = 6379, # FalkorDB port (default to 6379)
+    username = "", # FalkorDB username (empty by default)
+    password = "", # FalkorDB password (empty by default)
+    indexed_label = "Content", # The label of the indexed nodes (default to Content)
+    wipe_on_start = False, # Whether or not to wipe the hybridstore at start (default to False)
+)
+
+tools = [
+    ReadFile(
+        agent_state = agent_state,
+        filesystem = filesystem,
+    ),
+    WriteFile(
+        agent_state = agent_state,
+        filesystem = filesystem
+    ),
+    AppendFile(
+        agent_state = agent_state,
+        filesystem = filesystem,
+    ),
+    DocumentSearch(
+        filesystem = filesystem,
+    ),
+    InternalShell(
+        agent_state = agent_state,
+        filesystem = filesystem,
+    ),
+    Upload(
+        filesystem = filesystem,
+        # Where the system will save the zip archive (default to current working directory)
+        downloads_directory = "~/Downloads",
+    ),
+]
+
+agent = GraphProgramInterpreter(
+    program_memory = program_memory,
+    agent_state = agent_state,
+    tools = tools,
+)
+```

docs/api/hybridstores/program-memory.md

@@ -0,0 +1,65 @@
+---
+sidebar_position: 1
+---
+
+# Program Memory
+## HybridAGI program store
+
+The Program Memory is the only mandatory memory for our agent system. It is responsible for storing the dependency tree of the Cypher graph programs, along with the programs themselves. This memory enables the agent to read, write, and search for its own graph programs while safeguarding the main program and its dependencies from being modified by the system.
+
+<figure>
+  <p align="center">
+    ![HybridAGI program memory](img/program_memory.png)
+    <figcaption align="center"><b>Fig.1 - HybridAGI's program memory implements the dependency tree of its prompt programs. This structure protects the main prompt program from being modified by the Agent system.</b></figcaption>
+  </p>
+</figure>
+
+## Usage
+
+```python
+from hybridagi import ProgramMemory
+from hybridagi import SentenceTransformerEmbeddings
+from hybridagi.tools import (
+    ReadProgram,
+    WriteProgram,
+    CallProgram,
+    ProgramSearch,
+)
+
+embeddings = SentenceTransformerEmbeddings(
+    dim = 384,
+    model_name_or_path = "sentence-transformers/all-MiniLM-L6-v2",
+)
+
+program_memory = ProgramMemory(
+    index_name = "hybrid_agi", # The global index
+    embeddings = embeddings, # The embeddings to use
+    graph_index = "program_memory", # The hybridstore index (default to program_memory)
+    hostname = "localhost", # FalkorDB hostname (default to localhost)
+    port = 6379, # FalkorDB port (default to 6379)
+    username = "", # FalkorDB username (empty by default)
+    password = "", # FalkorDB password (empty by default)
+    indexed_label = "Content", # The label of the indexed nodes (default to Content)
+    wipe_on_start = False, # Whether or not to wipe the hybridstore at start (default to False)
+)
+
+tools = [
+    ReadProgram(
+        program_memory = program_memory
+    ),
+    WriteProgram(
+        program_memory = program_memory,
+    ),
+    CallProgram(
+        program_memory = program_memory,
+    ),
+    ProgramSearch(
+        program_memory = program_memory,
+    ),
+]
+
+interpreter = GraphProgramInterpreter(
+    program_memory = program_memory,
+    tools = tools,
+)
+```