3. Executing the Version Handshake.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finishing the Handshake\n",
    "\n",
    "Take a peek at [where we left off last time](http://localhost:8888/notebooks/2.%20Reading%20Version%20Messages.ipynb#Parsing-a-complete-Version-response).\n",
    "\n",
    "We were able to entice a response from our peer and then decode it. However, three problems remain:\n",
    "1. Our initial `version` message payload is hardcoded. We should be able to construct it using any parameters we like.\n",
    "2. After receiving our peer's `version` response, we don't listen for their `verack` response as the [version handshake](https://en.bitcoin.it/wiki/Version_Handshake) says we should.\n",
    "3. We don't send our `verack` upon receipt of our peer's `verack`, the final step in the handshake.\n",
    "\n",
    "Once we fix all these problems our program will be able to join the Bitcoin peer-to-peer network just like a [Bitcoin Core](https://github.com/bitcoin/bitcoin) full node does.  We won't be able to participate nearly as fully or effectively as Bitcoin Core node, but it's a start!\n",
    "\n",
    "The last 2 problems simple fixes, since the `Verack` message [is so simple](https://en.bitcoin.it/wiki/Protocol_documentation#verack). \n",
    "\n",
    "Problem #1 will be more involved. We've spent a lot of time learning to _deserialize_ Bitcoin network messages -- to turn raw bytes into Python objects. The 3rd problem demands we do the opposite: _serialize_ Bitcoin messages, to turn Python objects into raw bytes that can be sent over the network to our peers, who may not even have Python installed!\n",
    "\n",
    "If you've ever done any web development I'm sure you've learned to serialize and deserialize JSON, which is the de facto data representation in this arena. Bitcoin is no different, but it uses raw bytes instead of JSON. Pretty simple.\n",
    "\n",
    "Let's tackle these problems one-by-one. But first, some imports to set up the notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this loads a jupyter extension which allows us to reimport python\n",
    "# files every time we edit them\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "# import the code from last time\n",
    "# FIXME: these should be loaded from ibd.two.complete.\n",
    "# when API stabilizes I will replace ibd.two.complete with ibd.three.complete\n",
    "from ibd.three.complete import *\n",
    "\n",
    "# import all libraries we will need\n",
    "import os, time, socket, ipytest, pytest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Problem #1: Constructing Version Messages\n",
    "\n",
    "We want to be able to do something like this:\n",
    "\n",
    "```python\n",
    "services = 1  # just NODE_NETWORK ...\n",
    "my_address = \"7.7.7.7\"\n",
    "peer_address = \"9.9.9.9\"\n",
    "ver_msg = VersionMessage(\n",
    "    version=70015,\n",
    "    services=services,\n",
    "    time=time.time(),\n",
    "    addr_from=my_address,\n",
    "    addr_recv=peer_address,\n",
    "    nonce=1234567890,\n",
    "    user_agent=\"bitcoin-corps\",\n",
    "    start_height=0,\n",
    "    relay=1,\n",
    ")\n",
    "version_packet = Packet(\n",
    "    command=version_message.command, \n",
    "    payload=version_message.to_bytes()\n",
    ")\n",
    "packet_bytes = version_packet.to_bytes()\n",
    "\n",
    "sock = socket()\n",
    "sock.connect((peer_address, 8333))\n",
    "sock.send(packet_bytes)\n",
    "print(Packet.from_socket(sock))\n",
    "```\n",
    "\n",
    "This would do the same exact thing as [the last cell in lesson 2](http://localhost:8888/notebooks/2.%20Reading%20Version%20Messages.ipynb#Parsing-a-complete-Version-response), but the version message we send is no longer hard-coded.\n",
    "\n",
    "We are now free to send our peer whatever `version` number we like -- here we're choosing the most recent Bitcoin protocol version number 70015. We can advertise whatever `services` we like. We can define our own custom `user_agent` designating the Bitcoin implementation we're using: `bitcoin-corps`. And we can tell them we haven't started syncing the blockchain yet: `start_height=0`.\n",
    "\n",
    "Most of the above snippet already works. But the two serialization methods do not: \n",
    "1. `Version.to_bytes()`\n",
    "2. `Packet.to_bytes()`\n",
    "\n",
    "These will be somewhat analagous to the `Packet.from_socket()` and `Version.from_bytes()` methods we wrote previously: but they will do exactly the inverse operations. \n",
    "\n",
    "`Packet.from_socket` loads a Python `Packet` class instance from bytes read from a socket, but `Packet.to_bytes` takes a Python `Packet` class and converts it into a `bytes`-representation which we can send to our peer using `socket.send`. \n",
    "\n",
    "Similarly, `Version.from_bytes` takes the `payload` bytes of a `Packet` instance and turns it into a Python `Version` class, and `Version.to_bytes` will take a `Version` instance and turn it into `bytes`-representation in order to include it as the `payload` of an outgoing `Packet`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A Simplified Example\n",
    "\n",
    "Let's pretend the Bitcoin network also has a `pet` data type, similar to the `net_addr` and `services` we dealt with previously. Here's what the corresponding table would look like in the protocol documentation:\n",
    "\n",
    "\n",
    "| Field Size | Description | Data type | Comments                     |\n",
    "| ---------- | ----------- | --------- | ---------------------------- |\n",
    "| 3          | kind        | char[3]   | 'dog', 'cat', 'cow', or 'pig'|\n",
    "| 10         | name        | char[10]  | The pet's name               |\n",
    "\n",
    "Pretty simple, right. Two attributes: 3-character `kind` and 10-character `name`.\n",
    "\n",
    "Let's say we have a class like the one below, which already has a `from_bytes` classmethod defined to instantiate `Pet` instances from serialized `bytes` we receive over the wire. This leads to our first exercise ...\n",
    "\n",
    "### Exercise #X - Write `Pet.to_bytes`\n",
    "\n",
    "This would allow us to create an instance of our own `Pet`, serialize it into `bytes` and send it across the Bitcoin network (cringe, I know ...).\n",
    "\n",
    "If you look at the test, you can tell this method is correct if `pet_bytes == Pet.from_bytes(pet_bytes).to_bytes()`. That is, you should be able to turn it from bytes into a Python class and then back to bytes and have the very same bytes you started with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Pet:\n",
    "    valid_kinds = [b\"cat\", b\"dog\", b\"pig\", b\"cow\"]\n",
    "    \n",
    "    def __init__(self, kind, name):\n",
    "        self.kind = kind\n",
    "        self.name = name\n",
    "    \n",
    "    @classmethod\n",
    "    def from_bytes(cls, b):\n",
    "        stream = io.BytesIO(b)\n",
    "        kind = stream.read(3)\n",
    "        if kind not in cls.valid_kinds:\n",
    "            raise RuntimeError(\"invalid 'kind'\")\n",
    "        name = stream.read(10)\n",
    "        return cls(kind, name)\n",
    "    \n",
    "    def to_bytes(self):\n",
    "        raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_pet_to_bytes():\n",
    "    pet_bytes = b'pigbuddy'\n",
    "    pet = Pet.from_bytes(pet_bytes)\n",
    "    assert pet_bytes == pet.to_bytes()\n",
    "    \n",
    "ipytest.run_tests(doctest=True)\n",
    "ipytest.clean_test(\"test_pet_to_bytes*\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Packet.to_bytes\n",
    "\n",
    "So this is what `Packet.to_bytes` will look like.\n",
    "\n",
    "With this we just need to `int_to_bytes` and `command_to_bytes`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FIXME these should be exercises\n",
    "def encode_command(cmd):\n",
    "    padding_needed = 12 - len(cmd)\n",
    "    padding = b\"\\x00\" * padding_needed\n",
    "    return cmd + padding\n",
    "\n",
    "\n",
    "def int_to_bytes(i, length, byte_order=\"little\"):\n",
    "    return int.to_bytes(i, length, byte_order)\n",
    "\n",
    "\n",
    "NETWORK_MAGIC = 0xD9B4BEF9\n",
    "\n",
    "\n",
    "class Packet:\n",
    "    def __init__(self, command, payload):\n",
    "        self.command = command\n",
    "        self.payload = payload\n",
    "\n",
    "    @classmethod\n",
    "    def from_socket(cls, sock):\n",
    "        magic = read_magic(sock)\n",
    "        if magic != NETWORK_MAGIC:\n",
    "            raise RuntimeError(f'Network magic \"{magic}\" is wrong')\n",
    "\n",
    "        command = read_command(sock)\n",
    "        payload_length = read_length(sock)\n",
    "        checksum = read_checksum(sock)\n",
    "        payload = read_payload(sock, payload_length)\n",
    "\n",
    "        computed_checksum = compute_checksum(payload)\n",
    "        if computed_checksum != checksum:\n",
    "            raise RuntimeError(\"Checksums don't match\")\n",
    "\n",
    "        if payload_length != len(payload):\n",
    "            raise RuntimeError(\n",
    "                \"Tried to read {payload_length} bytes, only received {len(payload)} bytes\"\n",
    "            )\n",
    "\n",
    "        return cls(command, payload)\n",
    "\n",
    "    def to_bytes(self):\n",
    "        result = int_to_bytes(NETWORK_MAGIC, 4)\n",
    "        result += encode_command(self.command)\n",
    "        result += int_to_bytes(len(self.payload), 4)\n",
    "        result += compute_checksum(self.payload)\n",
    "        result += self.payload\n",
    "        return result\n",
    "\n",
    "    def __repr__(self):\n",
    "        return f\"<Packet command={self.command}>\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Version.to_bytes\n",
    "\n",
    "define all helper functions\n",
    "\n",
    "write unittests to test each line of to_bytes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def time_to_bytes(t, num_bytes):\n",
    "    return int_to_bytes(t, num_bytes)\n",
    "\n",
    "def str_to_var_str(s):\n",
    "    pass\n",
    "\n",
    "def bool_to_bytes(b):\n",
    "    pass\n",
    "\n",
    "class VersionMessage:\n",
    "\n",
    "    command = b\"version\"\n",
    "\n",
    "    def __init__(self,\n",
    "        version,\n",
    "        services,\n",
    "        time,\n",
    "        addr_recv,\n",
    "        addr_from,\n",
    "        nonce,\n",
    "        user_agent,\n",
    "        start_height,\n",
    "        relay,\n",
    "    ):\n",
    "        self.version = version\n",
    "        self.services = services\n",
    "        self.time = time\n",
    "        self.addr_recv = addr_recv\n",
    "        self.addr_from = addr_from\n",
    "        self.nonce = nonce\n",
    "        self.user_agent = user_agent\n",
    "        self.start_height = start_height\n",
    "        self.relay = relay\n",
    "\n",
    "    @classmethod\n",
    "    def from_bytes(cls, payload):\n",
    "        stream = io.BytesIO(payload)\n",
    "        version = read_int(stream, 4)\n",
    "        services = read_services(stream)\n",
    "        time = read_time(stream)\n",
    "        addr_recv = Address.from_stream(stream, version_msg=True)\n",
    "        addr_from = Address.from_stream(stream, version_msg=True)\n",
    "        nonce = read_int(stream, 8)\n",
    "        user_agent = read_var_str(stream)\n",
    "        start_height = read_int(stream, 4)\n",
    "        relay = read_bool(stream)\n",
    "        return cls(\n",
    "            version,\n",
    "            services,\n",
    "            time,\n",
    "            addr_recv,\n",
    "            addr_from,\n",
    "            nonce,\n",
    "            user_agent,\n",
    "            start_height,\n",
    "            relay,\n",
    "        )\n",
    "    \n",
    "    def to_bytes(self):\n",
    "        msg = int_to_bytes(self.version, 4)\n",
    "        msg += services_to_bytes(self.services)\n",
    "        msg += time_to_bytes(self.time, 8)\n",
    "        msg += self.addr_recv.to_bytes()  # TODO\n",
    "        msg += self.addr_from.to_bytes()\n",
    "        msg += int_to_bytes(self.nonce, 8)\n",
    "        msg += str_to_var_str(self.user_agent)\n",
    "        msg += int_to_bytes(self.start_height, 4)\n",
    "        msg += bool_to_bytes(self.relay)\n",
    "        return msg\n",
    "    \n",
    "    def __repr__(self):\n",
    "        return f\"<Message command={self.command}>\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: implement `bool_to_bytes`\n",
    "\n",
    "Hint: turn the argument `b` into an integer, then use `int_to_bytes` to serialize it into a 1 byte bytestring."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bool_to_bytes(b): \n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_bool_to_bytes():\n",
    "    booleans = [\n",
    "        True,\n",
    "        False\n",
    "    ]\n",
    "    answers = [\n",
    "        b'\\x01',\n",
    "        b'\\x00',\n",
    "    ]\n",
    "    for boolean, answer in zip(booleans, answers):\n",
    "        assert answer == bool_to_bytes(boolean)\n",
    "        \n",
    "ipytest.run_tests(doctest=True)\n",
    "ipytest.clean_tests(\"test_bool_to_bytes*\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to implement `str_to_var_str`, which is required to serialize the user-agent, we first need to implement `int_to_var_int` because `str_to_var_str` uses it to encode the length of the variable-length string.\n",
    "\n",
    "### Exercise: implement int_to_var_int\n",
    "\n",
    "[reference](https://en.bitcoin.it/wiki/Protocol_documentation#Variable_length_integer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def int_to_var_int(i):\n",
    "    if i < 0xfd:\n",
    "        return bytes([i])\n",
    "    elif i <= 0xffff:\n",
    "        return b\"\\xfd\" + \"FIXME\"\n",
    "    elif i <= 0xffffffff:\n",
    "        \"FIXME\"\n",
    "    elif \"FIXME\":\n",
    "        \"FIXME\"\n",
    "    else:\n",
    "        raise RuntimeError(\"integer too large: {}\".format(i))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FIXME: we should have 4 separate tests ...\n",
    "# So student can make one little part pass at a time\n",
    "\n",
    "def test_int_to_var_int():\n",
    "    numbers = [\n",
    "        0x10,\n",
    "        0x1000,\n",
    "        0x10000000,\n",
    "        0x1000000000000000\n",
    "    ]\n",
    "    answers = [\n",
    "        b'\\x10',\n",
    "        b'\\xfd\\x00\\x10',\n",
    "        b'\\xfe\\x00\\x00\\x00\\x10',\n",
    "        b'\\xff\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x10',\n",
    "    ]\n",
    "    for number, answer in zip(numbers, answers):\n",
    "        assert answer == int_to_var_int(number)\n",
    "\n",
    "ipytest.run_tests(doctest=True)\n",
    "ipytest.clean_tests(\"test_int_to_var_int*\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: Implement `str_to_var_str`\n",
    "\n",
    "[reference](https://en.bitcoin.it/wiki/Protocol_documentation#Variable_length_string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def str_to_var_str(s):\n",
    "    length = len(s)\n",
    "    return int_to_var_int(length) + s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_str_to_var_str():\n",
    "    strings = [\n",
    "        b\"x\" * 0x10,\n",
    "        b\"x\" * 0x1000,\n",
    "        b\"x\" * 0x10000000,\n",
    "        b\"x\" * 0x100000001,\n",
    "    ]\n",
    "    answers = [\n",
    "        b\"\\x10\" + strings[0],\n",
    "        b\"\\xfd\" + b\"\\x00\\x10\" + strings[1], # FIXME doesn't seem right ...\n",
    "#         b\"\\xfe\" + b\"\\x00\\x10\" + strings[2],\n",
    "#         b\"\\xff\" + strings[3],\n",
    "    ]\n",
    "    for string, answer in zip(strings, answers):\n",
    "        print(answer)\n",
    "        print(str_to_var_str(string))\n",
    "        assert answer == str_to_var_str(string)\n",
    "\n",
    "ipytest.run_tests(doctest=True)\n",
    "ipytest.clean_tests(\"test_str_to_var_str*\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Address.to_bytes()\n",
    "\n",
    "don't give them anything\n",
    "\n",
    "First test implements port_to_bytes\n",
    "\n",
    "Next test checks `Address.to_bytes` line by line / attribute by attribute ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# FIXME these should be exercises\n",
    "\n",
    "def port_to_bytes(port):\n",
    "    return int_to_bytes(port, 2, byte_order=\"big\")\n",
    "\n",
    "IPV4_PREFIX = b\"\\x00\" * 10 + b\"\\xff\" * 2\n",
    "\n",
    "def ip_to_bytes(ip):\n",
    "    if \":\" in ip:  # determine if address is IPv6\n",
    "        return socket.inet_pton(socket.AF_INET6, ip)\n",
    "    else:\n",
    "        return IPV4_PREFIX + socket.inet_pton(socket.AF_INET, ip)\n",
    "\n",
    "class Address:\n",
    "    def __init__(self, services, ip, port, time):\n",
    "        self.services = services\n",
    "        self.ip = ip\n",
    "        self.port = port\n",
    "        self.time = time\n",
    "\n",
    "    @classmethod\n",
    "    def from_bytes(cls, bytes_, version_msg=False):\n",
    "        stream = io.BytesIO(bytes_)\n",
    "        return cls.from_stream(stream, version_msg)\n",
    "\n",
    "    @classmethod\n",
    "    def from_stream(cls, stream, version_msg=False):\n",
    "        if version_msg:\n",
    "            time = None\n",
    "        else:\n",
    "            time = read_time(stream)\n",
    "        services = read_services(stream)\n",
    "        ip = read_ip(stream)\n",
    "        port = read_port(stream)\n",
    "        return cls(services, ip, port, time)\n",
    "\n",
    "    def to_bytes(self, version_msg=False):\n",
    "        result = b\"\"\n",
    "        if self.time:\n",
    "            result += time_to_bytes(self.time, 4)\n",
    "        result += services_to_bytes(self.services)\n",
    "        result += ip_to_bytes(self.ip)\n",
    "        result += port_to_bytes(self.port)\n",
    "        return result\n",
    "\n",
    "    def __eq__(self, other):\n",
    "        return self.__dict__ == other.__dict__\n",
    "\n",
    "    def __repr__(self):\n",
    "        return f\"<Address {self.ip}:{self.port}>\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Constructing and Serializing a Version Message\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibd.three.complete import *  # get the final version ...\n",
    "\n",
    "services = 1\n",
    "my_ip = \"7.7.7.7\"\n",
    "peer_ip = \"6.6.6.6\"\n",
    "port = 8333\n",
    "now = int(time.time())\n",
    "\n",
    "# addresses in version messages don't have \"time\" attributes\n",
    "my_address = Address(services, my_ip, port, time=None)\n",
    "peer_address = Address(services, peer_ip, port, time=None)\n",
    "\n",
    "version_message = VersionMessage(\n",
    "    version=70015,\n",
    "    services=services,\n",
    "    time=now,\n",
    "    # FIXME should we make this not in the first time this block of code appears?\n",
    "    # we're going to send this message, so it's \"from\" us ...\n",
    "    addr_from=my_address,\n",
    "    # and our peer will receive it\n",
    "    addr_recv=peer_address,\n",
    "    nonce=73948692739875,\n",
    "    user_agent=b\"bitcoin-corps\",\n",
    "    start_height=0,\n",
    "    relay=1,\n",
    ")\n",
    "\n",
    "payload = version_message.to_bytes()\n",
    "print(\"serialized version: \", payload)\n",
    "\n",
    "packet = Packet(command=version_message.command, payload=payload)\n",
    "print(\"serialized packet: \", packet.to_bytes())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Problem #2: Read Verack\n",
    "\n",
    "Verack messages [don't have payload](https://en.bitcoin.it/wiki/Protocol_documentation#verack). Remember all that work we had to do while implementing `Packet.from_socket` and `Vesion.from_stream`? We won't have to do anything of the sort while implementing our `Verack` class.\n",
    "\n",
    "For consistency sake we're going to maintain the conventions used previously:\n",
    "* class-level `command` attribute.\n",
    "* `from_bytes` classmethod\n",
    "* `__repr__` method, which dictates how the Verack class will be printed in [some circumstances](https://stackoverflow.com/questions/1436703/difference-between-str-and-repr).\n",
    "\n",
    "Here's the implementation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class VerackMessage:\n",
    "\n",
    "    command = b'verack'\n",
    "    \n",
    "    @classmethod\n",
    "    def from_bytes(cls, s):\n",
    "        return cls()\n",
    "    \n",
    "    def __repr__(self):\n",
    "        return \"<Verack>\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like I said, pretty simple!\n",
    "\n",
    "# Problem #3: Verack Response\n",
    "\n",
    "To send a `verack` response it would be sufficent to do the following:\n",
    "\n",
    "```python\n",
    "packet = Packet(command=b\"verack\", payload=b\"\")\n",
    "sock.send(packet.to_bytes())\n",
    "```\n",
    "\n",
    "But that would be inconsistent with the convention we set in the last section of creating `Packet` instances this way:\n",
    "\n",
    "```python\n",
    "packet = Packet(\n",
    "    command=some_message.command, \n",
    "    payload=some_message.to_bytes()\n",
    ")\n",
    "```\n",
    "\n",
    "And from now on we should avoid hard-coding anything we don't have to. So let's implement a `VerackMessage.to_bytes()` just like we did with `VersionMessage`. This will allow us to use the same calling when sending either kind of message."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class VerackMessage:\n",
    "\n",
    "    command = b'verack'\n",
    "\n",
    "    @classmethod\n",
    "    def from_bytes(cls, s):\n",
    "        return cls()\n",
    "\n",
    "    def to_bytes(self):\n",
    "        return b\"\"\n",
    "    \n",
    "    def __repr__(self):\n",
    "        return \"<Verack>\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Handshake "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import socket\n",
    "import time\n",
    "\n",
    "from ibd.three.complete import *  # get the final version ...\n",
    "\n",
    "\n",
    "def handshake(address):\n",
    "    # Arguments for our outgoing VersionMessage\n",
    "    services = 1\n",
    "    my_ip = \"7.7.7.7\"\n",
    "    peer_ip = address[0]\n",
    "    port = address[1]\n",
    "    now = int(time.time())\n",
    "    my_address = Address(services, my_ip, port, time=None)\n",
    "    peer_address = Address(services, peer_ip, port, time=None)\n",
    "\n",
    "    # Create out outgoing VersionMessage and Packet instances\n",
    "    version_message = VersionMessage(\n",
    "        version=70015,\n",
    "        services=services,\n",
    "        time=now,\n",
    "        addr_from=my_address,\n",
    "        addr_recv=peer_address,\n",
    "        nonce=73948692739875,\n",
    "        user_agent=b\"bitcoin-corps\",\n",
    "        start_height=0,\n",
    "        relay=1,\n",
    "    )\n",
    "    version_packet = Packet(\n",
    "        command=version_message.command, \n",
    "        payload=version_message.to_bytes()\n",
    "    )\n",
    "    serialized_packet = version_packet.to_bytes()\n",
    "\n",
    "    # Create the socket\n",
    "    sock = socket.socket()\n",
    "\n",
    "    # Initiate TCP connection\n",
    "    sock.connect(address)\n",
    "\n",
    "    # Initiate the Bitcoin version handshake\n",
    "    sock.send(serialized_packet)\n",
    "\n",
    "    # Receive their \"version\" response\n",
    "    pkt = Packet.from_socket(sock)\n",
    "    peer_version_message = VersionMessage.from_bytes(pkt.payload)\n",
    "    print(peer_version_message)\n",
    "\n",
    "    # Receive their \"verack\" response\n",
    "    pkt = Packet.from_socket(sock)\n",
    "    peer_verack_message = VerackMessage.from_bytes(pkt.payload)\n",
    "    print(peer_verack_message)\n",
    "\n",
    "    # Send out \"verack\" response\n",
    "    verack_message = VerackMessage()\n",
    "    verack_packet = Packet(verack_message.command, payload=verack_message.to_bytes())\n",
    "    sock.send(verack_packet.to_bytes())\n",
    "\n",
    "    return sock"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "handshake((\"35.198.151.21\", 8333))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What comes next?\n",
    "\n",
    "We've successfully executed the handshake, and our `handshake` function returns a live socket ...\n",
    "\n",
    "Why don't we just listen on the socket forever (or until the process is killedby typing \"ii\" or hitting the square \"stop\" button in the menu at the top of the screen) and see what happens?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sock = handshake((\"35.198.151.21\", 8333))\n",
    "\n",
    "while True:\n",
    "    packet = Packet.from_socket(sock)\n",
    "    print(packet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Like a Full Node\n",
    "\n",
    "You likely received all kinds of different command.\n",
    "\n",
    "Some -- like `feefilter` and `sendheaders` -- are your peer attempting to tell you what kind of data they want from you. You'll see a lot of these at the beginning of the output. But as time passes, you'll mostly see `inv` type messages. These are containers telling you about all kinds of new objects that your peer just found out about. If you were to decode these \"inv\" messages you could request the specific objects and you'd get a bunch of `tx` and some `block` messages in return\n",
    "\n",
    "FIXME should we do this?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Responding to inv messages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Following `addr` messages\n",
    "\n",
    "Another thing we can do is wait until our peer shares their list of currently connected addresses, and attempt to connect to some of their peers. Such a program would basically be a \"Bitcoin network crawler\", and building it will in fact be the topic of lesson 4.\n",
    "\n",
    "But to conclude lesson 3, let's build a simple, naive crawler and point our some deficiencies which will need to be corrected in order to realistically crawl the entire network.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from ibd.three.complete import *\n",
    "\n",
    " \n",
    "def simple_crawler():\n",
    "    addresses = [\n",
    "        (\"35.198.151.21\", 8333),\n",
    "        (\"91.221.70.137\", 8333),\n",
    "        (\"92.255.176.109\", 8333),\n",
    "        (\"94.199.178.17\", 8333),\n",
    "        (\"213.250.21.112\", 8333),\n",
    "    ]\n",
    "    while addresses:\n",
    "        \n",
    "        address = addresses.pop()\n",
    "        print('connecting to ', address)\n",
    "        sock = handshake(address)\n",
    "        \n",
    "        print(\"Waiting for addr message\")\n",
    "        listening = True\n",
    "        while listening:\n",
    "            packet = Packet.from_socket(sock)\n",
    "            if packet.command == b\"addr\":\n",
    "                addr_message = AddrMessage.from_bytes(packet.payload)\n",
    "                if len(addr_message.addresses) == 1 and addr_message.addresses[0].ip == address[0]:\n",
    "                    print(\"Received addr message with only our peer's address. Still waiting ...\")\n",
    "                else:\n",
    "                    print(f\"Received {len(addr_message.addresses)} addrs\")\n",
    "                    addresses.extend([(a.ip, a.port) for a in addr_message.addresses])\n",
    "                    listening = False\n",
    "    print(\"ran out of addresses. exiting.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "simple_crawler()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}