notebooks/api/0.8/01-submit-code.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Submitting Code to Syft as a Data Scientist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "SYFT_VERSION = \">=0.8.2.b0,<0.9\"\n",
    "package_string = f'\"syft{SYFT_VERSION}\"'\n",
    "# %pip install {package_string} -q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# third party\n",
    "import pandas as pd\n",
    "\n",
    "# syft absolute\n",
    "import syft as sy\n",
    "from syft.client.api import NodeIdentity\n",
    "from syft.service.request.request import RequestStatus\n",
    "\n",
    "sy.requires(SYFT_VERSION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Launch a Syft Domain Server"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Launch and connect to test-domain-1 server we setup in the previous notebook\n",
    "node = sy.orchestra.launch(name=\"test-domain-1\", port=\"auto\", dev_mode=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Every `node` exposes a \"guest\" client that allows some basic read operations on the node without creating an account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "guest_domain_client = node.client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Print this to see the few commands that are available for the guest client\n",
    "guest_domain_client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This will return the public credentials of the guest client\n",
    "guest_credentials = guest_domain_client.credentials"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Login into the Domain with Data Scientist credentials that we created in [00-load-data.ipynb](./00-load-data.ipynb) notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "jane_client = guest_domain_client.login(email=\"jane@caltech.edu\", password=\"abc123\")\n",
    "jane_client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "assert jane_client.credentials != guest_credentials"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explore available Syft Datasets in the Domain Node"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "results = jane_client.datasets.get_all()\n",
    "results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Test\n",
    "assert len(results) == 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use the \"Canada Trade Value\" dataset which is the first dataset in the list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dataset = results[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a Data Scientist, you can read the Mock dataset, but NOT the Private dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# access the mock data\n",
    "asset = dataset.assets[0]\n",
    "mock = asset.mock\n",
    "mock"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# cannot access the private data\n",
    "asset.data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test\n",
    "assert not isinstance(asset.data, pd.DataFrame)  # returns a permission error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can execute code on the Mock dataset!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "mock[\"Trade Value (US$)\"].sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "asset.id, asset.action_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a Syft Function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each Syft Function requires an Input & Output policy attached to the python function against which executions are verified.\n",
    "\n",
    "Syft provides the following default policies:\n",
    "* `sy.ExactMatch()` Input policy ensures that function executes against the exact inputs specified by Data Scientist.\n",
    "* `sy.OutputPolicyExecuteOnce()` Output policy makes sure that the Data Scientist can run the function only once against the input.\n",
    "\n",
    "We can also implement custom policies based on our requirements. (Refer to notebook [05-custom-policy](./05-custom-policy.ipynb) for more information.)\n",
    "\n",
    "For ease of use, Syft exposes a `@sy.syft_function_single_use()` decorator that will use `ExactMatch` input and `OutputPolicyExecuteOnce` output policies for the function.\n",
    "\n",
    "Let's go ahead and implement a function to perform some data analysis on the private dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# We wrap our compute function with this decorator to make the function run exactly on the `asset` dataset\n",
    "\n",
    "\n",
    "@sy.syft_function_single_use(trade_data=asset)\n",
    "def sum_trade_value_mil(trade_data):\n",
    "    # third party\n",
    "    import opendp.prelude as dp\n",
    "\n",
    "    dp.enable_features(\"contrib\")\n",
    "\n",
    "    aggregate = 0.0\n",
    "    base_lap = dp.m.make_base_laplace(\n",
    "        dp.atom_domain(T=float),\n",
    "        dp.absolute_distance(T=float),\n",
    "        scale=5.0,\n",
    "    )\n",
    "    noise = base_lap(aggregate)\n",
    "\n",
    "    df = trade_data\n",
    "    total = df[\"Trade Value (US$)\"].sum()\n",
    "    return (float(total / 1_000_000), float(noise))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we can run this, we need to run:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "pip install opendp\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can validate your code against the mock data, before submitting it to the Domain Server"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pointer = sum_trade_value_mil(trade_data=asset)\n",
    "result = pointer.get()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "assert result[0] == 9.738381"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "assert isinstance(result[1], float)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Tests\n",
    "assert len(sum_trade_value_mil.kwargs) == 1\n",
    "node_identity = NodeIdentity.from_api(jane_client.api)\n",
    "assert node_identity in sum_trade_value_mil.kwargs\n",
    "assert \"trade_data\" in sum_trade_value_mil.kwargs[node_identity]\n",
    "assert (\n",
    "    sum_trade_value_mil.input_policy_init_kwargs[node_identity][\"trade_data\"]\n",
    "    == asset.action_id\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(sum_trade_value_mil.code)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Submit your code to the Domain Server\n",
    "\n",
    "We start by creating new Syft Project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Create a new project\n",
    "new_project = sy.Project(\n",
    "    name=\"My Cool UN Project\",\n",
    "    description=\"Hi, I want to calculate the trade volume in million's with my cool code.\",\n",
    "    members=[jane_client],\n",
    ")\n",
    "new_project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Add a request to submit & execute the code\n",
    "result = new_project.create_code_request(sum_trade_value_mil, jane_client)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "assert len(jane_client.code.get_all()) == 1, str(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create the same code request with the exact same function should return an error\n",
    "result = new_project.create_code_request(sum_trade_value_mil, jane_client)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert len(jane_client.code.get_all()) == 1, str(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Once we start the project, it will submit the project along with the code request to the Domain Server\n",
    "project = new_project.send()\n",
    "project"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "assert isinstance(project, sy.service.project.project.Project), project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OR, when working on a project that already exists, we can simply use `<client>.get_project(name=\"project_name\")`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Or when working on a project that already exists\n",
    "project = jane_client.get_project(name=\"My Cool UN Project\")\n",
    "assert project\n",
    "assert len(project.events) == 1\n",
    "assert isinstance(project.events[0], sy.service.project.project.ProjectRequest)\n",
    "assert project.events[0].request.status == RequestStatus.PENDING"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running the Syft Function\n",
    "\n",
    "We can now execute our custom function by invoking the following"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "result = jane_client.code.sum_trade_value_mil(trade_data=asset)\n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "assert isinstance(result, sy.SyftError)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see that the result are not ready for the function, because it needs to approved by the Data Owners.\n",
    "\n",
    "Once approved, you can run the above cell again or go through [03-data-scientist-download-result](./03-data-scientist-download-result.ipynb) notebook for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Cleanup local domain server\n",
    "\n",
    "if node.node_type.value == \"python\":\n",
    "    node.land()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}