{ "cells": [ { "cell_type": "markdown", "id": "54a9fe493c76029b", "metadata": {}, "source": [ "# Walkthrough\n", "\n", "In this walkthrough, we'll create some graph computations with Pargraph and analyze NYC taxi data." ] }, { "cell_type": "markdown", "id": "5fc6423acb43c1a3", "metadata": {}, "source": [ "## Defining `delayed` functions\n", "\n", "`delayed` functions are pure, inseperable functions used to compose a graph computation. The pure nature of these functions allows them to be scheduled and executed concurrently across cores and machines.\n", "\n", "In a graph computation, `delayed` functions are the nodes and the edges represent the dependencies between `delayed` functions." ] }, { "cell_type": "markdown", "id": "5ca166291113d78b", "metadata": {}, "source": [ "### Example: NYC Yellow Taxi Fare Totals by Year\n", "\n", "The NYC Taxi and Limousine Commission publishes [data](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page) for every taxi trip since 2009. Each data file is in the [Parquet format](https://parquet.apache.org/) and contains every trip taken in a given month. Loading each file and processing them sequentially takes a very long time, so to speed it up we would like to load and process the files in parallel then perform an aggregation afterward.\n", "\n", "Here are some functions that we will need, defined as `delayed` functions:" ] }, { "cell_type": "code", "execution_count": 1, "id": "d825a8db8927436e", "metadata": { "ExecuteTime": { "end_time": "2025-04-14T14:58:05.358797Z", "start_time": "2025-04-14T14:58:04.781471Z" } }, "outputs": [], "source": [ "import pandas as pd\n", "from pargraph import delayed\n", "\n", "\n", "@delayed\n", "def get_total_amount_sum(file_url, unit):\n", " \"\"\"\n", " Calculate the sum of the 'total_amount' column from a Parquet file and divide it by a given unit.\n", "\n", " :param file_url: The URL of the Parquet file\n", " :param unit: The unit to divide the total amount sum by\n", " :return: The sum of the 'total_amount' column divided by the given unit\n", " \"\"\"\n", " return (\n", " pd.read_parquet(file_url, columns=[\"total_amount\"])[\"total_amount\"].sum() / unit\n", " )\n", "\n", "\n", "@delayed\n", "def add(*args):\n", " \"\"\"\n", " Calculate the sum of all the arguments provided.\n", "\n", " :param args: A variable number of arguments to be summed\n", " :return: The sum of all arguments\n", " \"\"\"\n", " return sum(args)\n", "\n", "\n", "@delayed\n", "def collect_result(*results):\n", " \"\"\"\n", " Collect results into a pandas Series, using every other element as the index and the remaining elements as the values.\n", "\n", " :param results: A variable number of arguments where even-indexed elements are used as the index and odd-indexed elements as the values\n", " :return: A pandas Series with the specified index and values\n", " \"\"\"\n", " return pd.Series(results[1::2], index=results[0::2])" ] }, { "cell_type": "markdown", "id": "1cf30e4a0a792c1a", "metadata": {}, "source": [ "A `delayed` function retains the same call semantics as a normal Python function:" ] }, { "cell_type": "code", "execution_count": 2, "id": "b52578dd17b2133f", "metadata": { "ExecuteTime": { "end_time": "2025-04-14T14:58:05.368478Z", "start_time": "2025-04-14T14:58:05.363805Z" } }, "outputs": [ { "data": { "text/plain": [ "3" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "add(1, 2)" ] }, { "cell_type": "markdown", "id": "f7376186fc74200c", "metadata": {}, "source": [ "## Defining `graph` functions\n", "\n", "`graph` functions compose a graph by calling `delayed` functions and other `graph` functions. `graph` functions are to `delayed` functions as molecules are to atoms." ] }, { "cell_type": "code", "execution_count": 3, "id": "18907642b86f7bbe", "metadata": { "ExecuteTime": { "end_time": "2025-04-14T14:58:05.554021Z", "start_time": "2025-04-14T14:58:05.550972Z" } }, "outputs": [], "source": [ "from pargraph import graph\n", "\n", "URL = \"https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_{year}-{month:02}.parquet\"\n", "\n", "\n", "@graph\n", "def nyc_yellow_taxi_fare_total_for_year(year, unit):\n", " \"\"\"\n", " Calculate the total fare amount for a given year by summing the total amounts from each month.\n", "\n", " :param year: The year for which to calculate the total fare amount\n", " :param unit: The unit to divide the total amount sum by\n", " :return: Total fare amount for the given year\n", " \"\"\"\n", " return add(\n", " *(\n", " get_total_amount_sum(URL.format(year=year, month=month), unit)\n", " for month in range(1, 13)\n", " )\n", " )" ] }, { "cell_type": "markdown", "id": "6ee53cde7282a3f6", "metadata": {}, "source": [ "A `graph` function retains the same call semantics as a normal Python function:" ] }, { "cell_type": "code", "execution_count": 4, "id": "c46b57bd724dc395", "metadata": { "ExecuteTime": { "end_time": "2025-04-14T14:58:58.042235Z", "start_time": "2025-04-14T14:58:05.564832Z" } }, "outputs": [ { "data": { "text/plain": [ "1145.8691257199998" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "nyc_yellow_taxi_fare_total_for_year(year=2024, unit=1_000_000)" ] }, { "cell_type": "markdown", "id": "76199eabe09723f3", "metadata": {}, "source": [ "## Converting to graphs\n", "\n", "Using `to_graph()`, we can generate a graph representation of the function." ] }, { "cell_type": "code", "execution_count": 5, "id": "932627b3af5e363a", "metadata": { "ExecuteTime": { "end_time": "2025-04-14T14:58:58.052201Z", "start_time": "2025-04-14T14:58:58.047317Z" } }, "outputs": [ { "data": { "text/plain": [ "Graph(consts={ConstKey(key='_2b4a8a0d1c4540bcba120bf22b548b33'): Const(type='msgpack', value='2U9odHRwczovL2QzN2NpNnZ6dXJ5Y2h4LmNsb3VkZnJvbnQubmV0L3RyaXAtZGF0YS95ZWxsb3dfdHJpcGRhdGFfMjAyNC0wMS5wYXJxdWV0'), ConstKey(key='_dfb19e5cdf664641a82537a3fb00746c'): Const(type='msgpack', value='2U9odHRwczovL2QzN2NpNnZ6dXJ5Y2h4LmNsb3VkZnJvbnQubmV0L3RyaXAtZGF0YS95ZWxsb3dfdHJpcGRhdGFfMjAyNC0wMi5wYXJxdWV0'), ConstKey(key='_1bab762fa4bf4f5cb2800bf3db7d2b75'): Const(type='msgpack', value='2U9odHRwczovL2QzN2NpNnZ6dXJ5Y2h4LmNsb3VkZnJvbnQubmV0L3RyaXAtZGF0YS95ZWxsb3dfdHJpcGRhdGFfMjAyNC0wMy5wYXJxdWV0'), ConstKey(key='_ff769f99997b406583c8ddbbc9d9dd0e'): Const(type='msgpack', value='2U9odHRwczovL2QzN2NpNnZ6dXJ5Y2h4LmNsb3VkZnJvbnQubmV0L3RyaXAtZGF0YS95ZWxsb3dfdHJpcGRhdGFfMjAyNC0wNC5wYXJxdWV0'), ConstKey(key='_75fb6201a22f46fc9876eb1735b63546'): Const(type='msgpack', value='2U9odHRwczovL2QzN2NpNnZ6dXJ5Y2h4LmNsb3VkZnJvbnQubmV0L3RyaXAtZGF0YS95ZWxsb3dfdHJpcGRhdGFfMjAyNC0wNS5wYXJxdWV0'), ConstKey(key='_2f8cff99612c4bd09c71e967241e7fbd'): Const(type='msgpack', value='2U9odHRwczovL2QzN2NpNnZ6dXJ5Y2h4LmNsb3VkZnJvbnQubmV0L3RyaXAtZGF0YS95ZWxsb3dfdHJpcGRhdGFfMjAyNC0wNi5wYXJxdWV0'), ConstKey(key='_990a0d33f4c44747818dc5806943a1b1'): Const(type='msgpack', value='2U9odHRwczovL2QzN2NpNnZ6dXJ5Y2h4LmNsb3VkZnJvbnQubmV0L3RyaXAtZGF0YS95ZWxsb3dfdHJpcGRhdGFfMjAyNC0wNy5wYXJxdWV0'), ConstKey(key='_91614fbdd23841fb9186e45fdadc61c0'): Const(type='msgpack', value='2U9odHRwczovL2QzN2NpNnZ6dXJ5Y2h4LmNsb3VkZnJvbnQubmV0L3RyaXAtZGF0YS95ZWxsb3dfdHJpcGRhdGFfMjAyNC0wOC5wYXJxdWV0'), ConstKey(key='_179683cff2b94209a7097645c2587b49'): Const(type='msgpack', value='2U9odHRwczovL2QzN2NpNnZ6dXJ5Y2h4LmNsb3VkZnJvbnQubmV0L3RyaXAtZGF0YS95ZWxsb3dfdHJpcGRhdGFfMjAyNC0wOS5wYXJxdWV0'), ConstKey(key='_f2cf83d1721741aabfe3b0dce420379e'): Const(type='msgpack', value='2U9odHRwczovL2QzN2NpNnZ6dXJ5Y2h4LmNsb3VkZnJvbnQubmV0L3RyaXAtZGF0YS95ZWxsb3dfdHJpcGRhdGFfMjAyNC0xMC5wYXJxdWV0'), ConstKey(key='_fbfe7b630c374bdd93c911d743dcf7eb'): Const(type='msgpack', value='2U9odHRwczovL2QzN2NpNnZ6dXJ5Y2h4LmNsb3VkZnJvbnQubmV0L3RyaXAtZGF0YS95ZWxsb3dfdHJpcGRhdGFfMjAyNC0xMS5wYXJxdWV0'), ConstKey(key='_9f13bd84b7f240cdb31d67408fc69185'): Const(type='msgpack', value='2U9odHRwczovL2QzN2NpNnZ6dXJ5Y2h4LmNsb3VkZnJvbnQubmV0L3RyaXAtZGF0YS95ZWxsb3dfdHJpcGRhdGFfMjAyNC0xMi5wYXJxdWV0')}, inputs={InputKey(key='unit'): None}, nodes={NodeKey(key='get_total_amount_sum_2d8dce044dc94629ae978f471e9a4b27'): FunctionCall(function=, args={'file_url': ConstKey(key='_2b4a8a0d1c4540bcba120bf22b548b33'), 'unit': InputKey(key='unit')}), NodeKey(key='get_total_amount_sum_5fd6d6d8a5f44649ad9e18325fa1d5fe'): FunctionCall(function=, args={'file_url': ConstKey(key='_dfb19e5cdf664641a82537a3fb00746c'), 'unit': InputKey(key='unit')}), NodeKey(key='get_total_amount_sum_6513415006b942bdbad75203aed04e59'): FunctionCall(function=, args={'file_url': ConstKey(key='_1bab762fa4bf4f5cb2800bf3db7d2b75'), 'unit': InputKey(key='unit')}), NodeKey(key='get_total_amount_sum_dc205da7c5cc4059b8b10ba4787de141'): FunctionCall(function=, args={'file_url': ConstKey(key='_ff769f99997b406583c8ddbbc9d9dd0e'), 'unit': InputKey(key='unit')}), NodeKey(key='get_total_amount_sum_4ce85af8e4c84cdfabe19a05691dda9a'): FunctionCall(function=, args={'file_url': ConstKey(key='_75fb6201a22f46fc9876eb1735b63546'), 'unit': InputKey(key='unit')}), NodeKey(key='get_total_amount_sum_d8553e95e0ca4b1390f240921abd0007'): FunctionCall(function=, args={'file_url': ConstKey(key='_2f8cff99612c4bd09c71e967241e7fbd'), 'unit': InputKey(key='unit')}), NodeKey(key='get_total_amount_sum_6620f973d25a461580614a2e99685a93'): FunctionCall(function=, args={'file_url': ConstKey(key='_990a0d33f4c44747818dc5806943a1b1'), 'unit': InputKey(key='unit')}), NodeKey(key='get_total_amount_sum_8487ed4445f6459b909e501d9e4bd0aa'): FunctionCall(function=, args={'file_url': ConstKey(key='_91614fbdd23841fb9186e45fdadc61c0'), 'unit': InputKey(key='unit')}), NodeKey(key='get_total_amount_sum_6661917548034435b9693327fdb97623'): FunctionCall(function=, args={'file_url': ConstKey(key='_179683cff2b94209a7097645c2587b49'), 'unit': InputKey(key='unit')}), NodeKey(key='get_total_amount_sum_a4af758ab3844b63b7de85fff5a3172a'): FunctionCall(function=, args={'file_url': ConstKey(key='_f2cf83d1721741aabfe3b0dce420379e'), 'unit': InputKey(key='unit')}), NodeKey(key='get_total_amount_sum_23ffa717da794003984e34ecb82d802a'): FunctionCall(function=, args={'file_url': ConstKey(key='_fbfe7b630c374bdd93c911d743dcf7eb'), 'unit': InputKey(key='unit')}), NodeKey(key='get_total_amount_sum_1feb5a75eeca467e933239c5d3d4f732'): FunctionCall(function=, args={'file_url': ConstKey(key='_9f13bd84b7f240cdb31d67408fc69185'), 'unit': InputKey(key='unit')}), NodeKey(key='add_eff00a44974644dfb8071545ddb5dd37'): FunctionCall(function=, args={'0': NodeOutputKey(key='get_total_amount_sum_2d8dce044dc94629ae978f471e9a4b27', output='result'), '1': NodeOutputKey(key='get_total_amount_sum_5fd6d6d8a5f44649ad9e18325fa1d5fe', output='result'), '2': NodeOutputKey(key='get_total_amount_sum_6513415006b942bdbad75203aed04e59', output='result'), '3': NodeOutputKey(key='get_total_amount_sum_dc205da7c5cc4059b8b10ba4787de141', output='result'), '4': NodeOutputKey(key='get_total_amount_sum_4ce85af8e4c84cdfabe19a05691dda9a', output='result'), '5': NodeOutputKey(key='get_total_amount_sum_d8553e95e0ca4b1390f240921abd0007', output='result'), '6': NodeOutputKey(key='get_total_amount_sum_6620f973d25a461580614a2e99685a93', output='result'), '7': NodeOutputKey(key='get_total_amount_sum_8487ed4445f6459b909e501d9e4bd0aa', output='result'), '8': NodeOutputKey(key='get_total_amount_sum_6661917548034435b9693327fdb97623', output='result'), '9': NodeOutputKey(key='get_total_amount_sum_a4af758ab3844b63b7de85fff5a3172a', output='result'), '10': NodeOutputKey(key='get_total_amount_sum_23ffa717da794003984e34ecb82d802a', output='result'), '11': NodeOutputKey(key='get_total_amount_sum_1feb5a75eeca467e933239c5d3d4f732', output='result')})}, outputs={OutputKey(key='result'): NodeOutputKey(key='add_eff00a44974644dfb8071545ddb5dd37', output='result')})" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "nyc_yellow_taxi_fare_total_for_year.to_graph(year=2024)" ] }, { "cell_type": "markdown", "id": "7f525eaa5185bece", "metadata": {}, "source": [ "Graphs can also be converted to a task graph using `to_dict()`. Notice that all arguments must be provided in order to convert the graph to a task graph:" ] }, { "cell_type": "code", "execution_count": 6, "id": "4897bf550f862808", "metadata": { "ExecuteTime": { "end_time": "2025-04-14T14:58:58.080269Z", "start_time": "2025-04-14T14:58:58.074571Z" } }, "outputs": [ { "data": { "text/plain": [ "({'const_Const_22bddfa577ea4c858eb8e0aeac5ca4e0': 'https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-01.parquet',\n", " 'const_Const_c37d2ea7a594424285bb160554d365a4': 'https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-02.parquet',\n", " 'const_Const_19b894a119d74cfab8520f32cbae8f75': 'https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-03.parquet',\n", " 'const_Const_d2eca83adc30467ba1b6d5cf7d46093e': 'https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-04.parquet',\n", " 'const_Const_7ae7fd2841234a5a8f7386b4788384e0': 'https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-05.parquet',\n", " 'const_Const_aab19f56cf3b4d51b3f0532b7485d536': 'https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-06.parquet',\n", " 'const_Const_736a1f5be50f42ab89a32325f9db50c0': 'https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-07.parquet',\n", " 'const_Const_104e0221d61945adb7f11268f0f9fdee': 'https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-08.parquet',\n", " 'const_Const_4f086769cc3245558798fc5b4fdb5d0f': 'https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-09.parquet',\n", " 'const_Const_ba852608eb694de0958d676b3b430940': 'https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-10.parquet',\n", " 'const_Const_e9690dfc980e443ca1f321a3f864458e': 'https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-11.parquet',\n", " 'const_Const_49f572270b10496fafd002f5f2344c45': 'https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-12.parquet',\n", " 'input_unit_bb96debe1c5f480f93589e836f4e156c': 1000000,\n", " 'node_output_get_total_amount_sum_6fee4a6518764247b8de2d249272810a': (,\n", " 'const_Const_22bddfa577ea4c858eb8e0aeac5ca4e0',\n", " 'input_unit_bb96debe1c5f480f93589e836f4e156c'),\n", " 'node_output_get_total_amount_sum_ae374f2e96004f84b95f55d8ac8a08ac': (,\n", " 'const_Const_c37d2ea7a594424285bb160554d365a4',\n", " 'input_unit_bb96debe1c5f480f93589e836f4e156c'),\n", " 'node_output_get_total_amount_sum_c40297479e064da39ca6ad7a615fa618': (,\n", " 'const_Const_19b894a119d74cfab8520f32cbae8f75',\n", " 'input_unit_bb96debe1c5f480f93589e836f4e156c'),\n", " 'node_output_get_total_amount_sum_0740847276ba4fa9be487c49d8a0e2da': (,\n", " 'const_Const_d2eca83adc30467ba1b6d5cf7d46093e',\n", " 'input_unit_bb96debe1c5f480f93589e836f4e156c'),\n", " 'node_output_get_total_amount_sum_b0848d68bd1043bebb788c6245fc7b51': (,\n", " 'const_Const_7ae7fd2841234a5a8f7386b4788384e0',\n", " 'input_unit_bb96debe1c5f480f93589e836f4e156c'),\n", " 'node_output_get_total_amount_sum_17ea5faef84c4864a801276eed9541a0': (,\n", " 'const_Const_aab19f56cf3b4d51b3f0532b7485d536',\n", " 'input_unit_bb96debe1c5f480f93589e836f4e156c'),\n", " 'node_output_get_total_amount_sum_bd00359f3d564e99856f322eb504db89': (,\n", " 'const_Const_736a1f5be50f42ab89a32325f9db50c0',\n", " 'input_unit_bb96debe1c5f480f93589e836f4e156c'),\n", " 'node_output_get_total_amount_sum_b8d1af4dbbad4c74918815f33f9d4909': (,\n", " 'const_Const_104e0221d61945adb7f11268f0f9fdee',\n", " 'input_unit_bb96debe1c5f480f93589e836f4e156c'),\n", " 'node_output_get_total_amount_sum_0ea7100e98c74dcd8b41ef79999be830': (,\n", " 'const_Const_4f086769cc3245558798fc5b4fdb5d0f',\n", " 'input_unit_bb96debe1c5f480f93589e836f4e156c'),\n", " 'node_output_get_total_amount_sum_f2c987ca4fd54ca18e3dc459dabd38cb': (,\n", " 'const_Const_ba852608eb694de0958d676b3b430940',\n", " 'input_unit_bb96debe1c5f480f93589e836f4e156c'),\n", " 'node_output_get_total_amount_sum_5de103642b9b406d9342d3a6b129f2da': (,\n", " 'const_Const_e9690dfc980e443ca1f321a3f864458e',\n", " 'input_unit_bb96debe1c5f480f93589e836f4e156c'),\n", " 'node_output_get_total_amount_sum_41bc34d152dd47459219c0639cfd7ca7': (,\n", " 'const_Const_49f572270b10496fafd002f5f2344c45',\n", " 'input_unit_bb96debe1c5f480f93589e836f4e156c'),\n", " 'node_output_add_a5836b630b1c433d898a2aedd7761e55': (,\n", " 'node_output_get_total_amount_sum_6fee4a6518764247b8de2d249272810a',\n", " 'node_output_get_total_amount_sum_ae374f2e96004f84b95f55d8ac8a08ac',\n", " 'node_output_get_total_amount_sum_c40297479e064da39ca6ad7a615fa618',\n", " 'node_output_get_total_amount_sum_0740847276ba4fa9be487c49d8a0e2da',\n", " 'node_output_get_total_amount_sum_b0848d68bd1043bebb788c6245fc7b51',\n", " 'node_output_get_total_amount_sum_17ea5faef84c4864a801276eed9541a0',\n", " 'node_output_get_total_amount_sum_bd00359f3d564e99856f322eb504db89',\n", " 'node_output_get_total_amount_sum_b8d1af4dbbad4c74918815f33f9d4909',\n", " 'node_output_get_total_amount_sum_0ea7100e98c74dcd8b41ef79999be830',\n", " 'node_output_get_total_amount_sum_f2c987ca4fd54ca18e3dc459dabd38cb',\n", " 'node_output_get_total_amount_sum_5de103642b9b406d9342d3a6b129f2da',\n", " 'node_output_get_total_amount_sum_41bc34d152dd47459219c0639cfd7ca7')},\n", " ['node_output_add_a5836b630b1c433d898a2aedd7761e55'])" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "nyc_yellow_taxi_fare_total_for_year.to_graph(year=2024).to_dict(unit=1_000_000)" ] }, { "cell_type": "markdown", "id": "d4052d90d9276d70", "metadata": {}, "source": [ "## Visualizing graphs\n", "\n", "The above outputs are hard to interpret and only particularly useful for a computer. Let's write a simple helper function to visualize the graphs as dot graphs." ] }, { "cell_type": "code", "execution_count": 7, "id": "3b992fdafda239bd", "metadata": { "ExecuteTime": { "end_time": "2025-04-14T14:58:58.100162Z", "start_time": "2025-04-14T14:58:58.097033Z" } }, "outputs": [], "source": [ "from IPython.display import SVG\n", "\n", "\n", "def get_graph_image(dot_graph):\n", " dot_graph.write_svg(\"temp.svg\")\n", " return SVG(\"temp.svg\")" ] }, { "cell_type": "markdown", "id": "2af693afb81efc05", "metadata": {}, "source": [ "Using `to_dot()`, we can generate a dot graph:" ] }, { "cell_type": "code", "execution_count": 8, "id": "dbc1b76e6302dad5", "metadata": { "ExecuteTime": { "end_time": "2025-04-14T14:58:58.506478Z", "start_time": "2025-04-14T14:58:58.111340Z" } }, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "G\n", "\n", "\n", "\n", "get_total_amount_sum_7b4d5b8b546f489fbc79d4d61f1adda5\n", "\n", "file_url\n", "\n", "unit\n", "\n", "get_total_amount_sum\n", "\n", "result\n", "\n", "\n", "\n", "add_4618897898834042ad6f4045a5cf7381\n", "\n", "args\n", "\n", "add\n", "\n", "result\n", "\n", "\n", "\n", "get_total_amount_sum_7b4d5b8b546f489fbc79d4d61f1adda5:outputs_result->add_4618897898834042ad6f4045a5cf7381:inputs_0\n", "\n", "\n", "\n", "\n", "\n", "const__51ec8314bcbb4487a323b227f32e3ae7\n", "\n", "https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-01.parquet\n", "\n", "\n", "\n", "const__51ec8314bcbb4487a323b227f32e3ae7->get_total_amount_sum_7b4d5b8b546f489fbc79d4d61f1adda5:inputs_file_url\n", "\n", "\n", "\n", "\n", "\n", "input_unit\n", "\n", "unit\n", "\n", "\n", "\n", "input_unit->get_total_amount_sum_7b4d5b8b546f489fbc79d4d61f1adda5:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_264379d6f8544afea33e1a312a2b492b\n", "\n", "file_url\n", "\n", "unit\n", "\n", "get_total_amount_sum\n", "\n", "result\n", "\n", "\n", "\n", "input_unit->get_total_amount_sum_264379d6f8544afea33e1a312a2b492b:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_9c9a50aaa1b84a0c95a2ca3b1d18a0d6\n", "\n", "file_url\n", "\n", "unit\n", "\n", "get_total_amount_sum\n", "\n", "result\n", "\n", "\n", "\n", "input_unit->get_total_amount_sum_9c9a50aaa1b84a0c95a2ca3b1d18a0d6:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_03b66ea51ac8437a86f464e3988514b1\n", "\n", "file_url\n", "\n", "unit\n", "\n", "get_total_amount_sum\n", "\n", "result\n", "\n", "\n", "\n", "input_unit->get_total_amount_sum_03b66ea51ac8437a86f464e3988514b1:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_405baccfe55c4f5997746950344c5e01\n", "\n", "file_url\n", "\n", "unit\n", "\n", "get_total_amount_sum\n", "\n", "result\n", "\n", "\n", "\n", "input_unit->get_total_amount_sum_405baccfe55c4f5997746950344c5e01:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_fa599adb569c42218fafac931f9141c6\n", "\n", "file_url\n", "\n", "unit\n", "\n", "get_total_amount_sum\n", "\n", "result\n", "\n", "\n", "\n", "input_unit->get_total_amount_sum_fa599adb569c42218fafac931f9141c6:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_19a54abe01474202852c614ac8a045e9\n", "\n", "file_url\n", "\n", "unit\n", "\n", "get_total_amount_sum\n", "\n", "result\n", "\n", "\n", "\n", "input_unit->get_total_amount_sum_19a54abe01474202852c614ac8a045e9:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_acba2c85605b4147925cd9977de56e88\n", "\n", "file_url\n", "\n", "unit\n", "\n", "get_total_amount_sum\n", "\n", "result\n", "\n", "\n", "\n", "input_unit->get_total_amount_sum_acba2c85605b4147925cd9977de56e88:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_b02cf983ee6a464686de87b3d1bebd1b\n", "\n", "file_url\n", "\n", "unit\n", "\n", "get_total_amount_sum\n", "\n", "result\n", "\n", "\n", "\n", "input_unit->get_total_amount_sum_b02cf983ee6a464686de87b3d1bebd1b:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_bb2a96ad3017494085ddc7a09ed23806\n", "\n", "file_url\n", "\n", "unit\n", "\n", "get_total_amount_sum\n", "\n", "result\n", "\n", "\n", "\n", "input_unit->get_total_amount_sum_bb2a96ad3017494085ddc7a09ed23806:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_ef6d7417475e403b9d95aa473ae7e6d5\n", "\n", "file_url\n", "\n", "unit\n", "\n", "get_total_amount_sum\n", "\n", "result\n", "\n", "\n", "\n", "input_unit->get_total_amount_sum_ef6d7417475e403b9d95aa473ae7e6d5:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_015401390ee24e4aa19f40cf1006fca3\n", "\n", "file_url\n", "\n", "unit\n", "\n", "get_total_amount_sum\n", "\n", "result\n", "\n", "\n", "\n", "input_unit->get_total_amount_sum_015401390ee24e4aa19f40cf1006fca3:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_264379d6f8544afea33e1a312a2b492b:outputs_result->add_4618897898834042ad6f4045a5cf7381:inputs_1\n", "\n", "\n", "\n", "\n", "\n", "const__6c7ab3efb258443dad24bdd177648b1c\n", "\n", "https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-02.parquet\n", "\n", "\n", "\n", "const__6c7ab3efb258443dad24bdd177648b1c->get_total_amount_sum_264379d6f8544afea33e1a312a2b492b:inputs_file_url\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_9c9a50aaa1b84a0c95a2ca3b1d18a0d6:outputs_result->add_4618897898834042ad6f4045a5cf7381:inputs_2\n", "\n", "\n", "\n", "\n", "\n", "const__13801412522c488b9034c9d27af180fe\n", "\n", "https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-03.parquet\n", "\n", "\n", "\n", "const__13801412522c488b9034c9d27af180fe->get_total_amount_sum_9c9a50aaa1b84a0c95a2ca3b1d18a0d6:inputs_file_url\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_03b66ea51ac8437a86f464e3988514b1:outputs_result->add_4618897898834042ad6f4045a5cf7381:inputs_3\n", "\n", "\n", "\n", "\n", "\n", "const__23ad884f016c4ce8821928b36c46bf3e\n", "\n", "https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-04.parquet\n", "\n", "\n", "\n", "const__23ad884f016c4ce8821928b36c46bf3e->get_total_amount_sum_03b66ea51ac8437a86f464e3988514b1:inputs_file_url\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_405baccfe55c4f5997746950344c5e01:outputs_result->add_4618897898834042ad6f4045a5cf7381:inputs_4\n", "\n", "\n", "\n", "\n", "\n", "const__e3e55f98dd2549b88803445b6b8e5d01\n", "\n", "https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-05.parquet\n", "\n", "\n", "\n", "const__e3e55f98dd2549b88803445b6b8e5d01->get_total_amount_sum_405baccfe55c4f5997746950344c5e01:inputs_file_url\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_fa599adb569c42218fafac931f9141c6:outputs_result->add_4618897898834042ad6f4045a5cf7381:inputs_5\n", "\n", "\n", "\n", "\n", "\n", "const__37688a2f73c04bec92ee8dca3a917c20\n", "\n", "https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-06.parquet\n", "\n", "\n", "\n", "const__37688a2f73c04bec92ee8dca3a917c20->get_total_amount_sum_fa599adb569c42218fafac931f9141c6:inputs_file_url\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_19a54abe01474202852c614ac8a045e9:outputs_result->add_4618897898834042ad6f4045a5cf7381:inputs_6\n", "\n", "\n", "\n", "\n", "\n", "const__2f77c9059fe54619b763739316f2ab92\n", "\n", "https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-07.parquet\n", "\n", "\n", "\n", "const__2f77c9059fe54619b763739316f2ab92->get_total_amount_sum_19a54abe01474202852c614ac8a045e9:inputs_file_url\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_acba2c85605b4147925cd9977de56e88:outputs_result->add_4618897898834042ad6f4045a5cf7381:inputs_7\n", "\n", "\n", "\n", "\n", "\n", "const__2585ac1fb91d4ba7888ef9739647c38a\n", "\n", "https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-08.parquet\n", "\n", "\n", "\n", "const__2585ac1fb91d4ba7888ef9739647c38a->get_total_amount_sum_acba2c85605b4147925cd9977de56e88:inputs_file_url\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_b02cf983ee6a464686de87b3d1bebd1b:outputs_result->add_4618897898834042ad6f4045a5cf7381:inputs_8\n", "\n", "\n", "\n", "\n", "\n", "const__549b25e07a6b47238e00852e00a39fc1\n", "\n", "https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-09.parquet\n", "\n", "\n", "\n", "const__549b25e07a6b47238e00852e00a39fc1->get_total_amount_sum_b02cf983ee6a464686de87b3d1bebd1b:inputs_file_url\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_bb2a96ad3017494085ddc7a09ed23806:outputs_result->add_4618897898834042ad6f4045a5cf7381:inputs_9\n", "\n", "\n", "\n", "\n", "\n", "const__0a23fbe64045476685c4a1866ecaf6b1\n", "\n", "https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-10.parquet\n", "\n", "\n", "\n", "const__0a23fbe64045476685c4a1866ecaf6b1->get_total_amount_sum_bb2a96ad3017494085ddc7a09ed23806:inputs_file_url\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_ef6d7417475e403b9d95aa473ae7e6d5:outputs_result->add_4618897898834042ad6f4045a5cf7381:inputs_10\n", "\n", "\n", "\n", "\n", "\n", "const__6b85f595cbd145e39434f4149ab8e820\n", "\n", "https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-11.parquet\n", "\n", "\n", "\n", "const__6b85f595cbd145e39434f4149ab8e820->get_total_amount_sum_ef6d7417475e403b9d95aa473ae7e6d5:inputs_file_url\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_015401390ee24e4aa19f40cf1006fca3:outputs_result->add_4618897898834042ad6f4045a5cf7381:inputs_11\n", "\n", "\n", "\n", "\n", "\n", "const__28e6e85a19174581a20740339a015e4e\n", "\n", "https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2024-12.parquet\n", "\n", "\n", "\n", "const__28e6e85a19174581a20740339a015e4e->get_total_amount_sum_015401390ee24e4aa19f40cf1006fca3:inputs_file_url\n", "\n", "\n", "\n", "\n", "\n", "_4218e029e8d94829a1aa9e2da788dbe4\n", "\n", "result\n", "\n", "\n", "\n", "add_4618897898834042ad6f4045a5cf7381:outputs_result->_4218e029e8d94829a1aa9e2da788dbe4\n", "\n", "\n", "\n", "\n", "" ], "text/plain": [ "" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "get_graph_image(nyc_yellow_taxi_fare_total_for_year.to_graph(year=2024).to_dot())" ] }, { "cell_type": "markdown", "id": "aa3883ddf38221db", "metadata": {}, "source": [ "If we only want to see the node relations, we can pass the arguments `no_input`, `no_const`, and `no_output` to make the graph less noisy:" ] }, { "cell_type": "code", "execution_count": 9, "id": "75f4bc450bf57f11", "metadata": { "ExecuteTime": { "end_time": "2025-04-14T14:58:58.900322Z", "start_time": "2025-04-14T14:58:58.515488Z" } }, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "G\n", "\n", "\n", "\n", "get_total_amount_sum_b518722c06cb42acae11cf55eba9f817\n", "\n", "file_url\n", "\n", "unit\n", "\n", "get_total_amount_sum\n", "\n", "result\n", "\n", "\n", "\n", "add_6a83beb923704119a7f98755db388fc8\n", "\n", "args\n", "\n", "add\n", "\n", "result\n", "\n", "\n", "\n", "get_total_amount_sum_b518722c06cb42acae11cf55eba9f817:outputs_result->add_6a83beb923704119a7f98755db388fc8:inputs_0\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_888685ce21904f55a5ea0fb36791b97d\n", "\n", "file_url\n", "\n", "unit\n", "\n", "get_total_amount_sum\n", "\n", "result\n", "\n", "\n", "\n", "get_total_amount_sum_888685ce21904f55a5ea0fb36791b97d:outputs_result->add_6a83beb923704119a7f98755db388fc8:inputs_1\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_1838141e4720400bb10ddc8c7340660c\n", "\n", "file_url\n", "\n", "unit\n", "\n", "get_total_amount_sum\n", "\n", "result\n", "\n", "\n", "\n", "get_total_amount_sum_1838141e4720400bb10ddc8c7340660c:outputs_result->add_6a83beb923704119a7f98755db388fc8:inputs_2\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_54a8fa418cc94018a17ad3fb70a1e63a\n", "\n", "file_url\n", "\n", "unit\n", "\n", "get_total_amount_sum\n", "\n", "result\n", "\n", "\n", "\n", "get_total_amount_sum_54a8fa418cc94018a17ad3fb70a1e63a:outputs_result->add_6a83beb923704119a7f98755db388fc8:inputs_3\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_f3817406aec94a6c83c1da67d9228c1c\n", "\n", "file_url\n", "\n", "unit\n", "\n", "get_total_amount_sum\n", "\n", "result\n", "\n", "\n", "\n", "get_total_amount_sum_f3817406aec94a6c83c1da67d9228c1c:outputs_result->add_6a83beb923704119a7f98755db388fc8:inputs_4\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_4b0ac5e237a741dcb7f0d4d0e2e54c9e\n", "\n", "file_url\n", "\n", "unit\n", "\n", "get_total_amount_sum\n", "\n", "result\n", "\n", "\n", "\n", "get_total_amount_sum_4b0ac5e237a741dcb7f0d4d0e2e54c9e:outputs_result->add_6a83beb923704119a7f98755db388fc8:inputs_5\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_752cc865ea53447fa3252a4fa1c3d1c0\n", "\n", "file_url\n", "\n", "unit\n", "\n", "get_total_amount_sum\n", "\n", "result\n", "\n", "\n", "\n", "get_total_amount_sum_752cc865ea53447fa3252a4fa1c3d1c0:outputs_result->add_6a83beb923704119a7f98755db388fc8:inputs_6\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_4eb779e6eb9741c08164ee316d39510c\n", "\n", "file_url\n", "\n", "unit\n", "\n", "get_total_amount_sum\n", "\n", "result\n", "\n", "\n", "\n", "get_total_amount_sum_4eb779e6eb9741c08164ee316d39510c:outputs_result->add_6a83beb923704119a7f98755db388fc8:inputs_7\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_b39bd96b21d64db08116126f86b5d5e9\n", "\n", "file_url\n", "\n", "unit\n", "\n", "get_total_amount_sum\n", "\n", "result\n", "\n", "\n", "\n", "get_total_amount_sum_b39bd96b21d64db08116126f86b5d5e9:outputs_result->add_6a83beb923704119a7f98755db388fc8:inputs_8\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_f7b4c8931d6c404ca93df2ffdb296475\n", "\n", "file_url\n", "\n", "unit\n", "\n", "get_total_amount_sum\n", "\n", "result\n", "\n", "\n", "\n", "get_total_amount_sum_f7b4c8931d6c404ca93df2ffdb296475:outputs_result->add_6a83beb923704119a7f98755db388fc8:inputs_9\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_8d047b2b32dd4837a1f47566ae3e209f\n", "\n", "file_url\n", "\n", "unit\n", "\n", "get_total_amount_sum\n", "\n", "result\n", "\n", "\n", "\n", "get_total_amount_sum_8d047b2b32dd4837a1f47566ae3e209f:outputs_result->add_6a83beb923704119a7f98755db388fc8:inputs_10\n", "\n", "\n", "\n", "\n", "\n", "get_total_amount_sum_0f10f19fd26a47afa32d22985fd4d507\n", "\n", "file_url\n", "\n", "unit\n", "\n", "get_total_amount_sum\n", "\n", "result\n", "\n", "\n", "\n", "get_total_amount_sum_0f10f19fd26a47afa32d22985fd4d507:outputs_result->add_6a83beb923704119a7f98755db388fc8:inputs_11\n", "\n", "\n", "\n", "\n", "" ], "text/plain": [ "" ] }, "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ "get_graph_image(\n", " nyc_yellow_taxi_fare_total_for_year.to_graph(year=2024).to_dot(\n", " no_input=True, no_const=True, no_output=True\n", " )\n", ")" ] }, { "cell_type": "markdown", "id": "d478be2ff1fc8893", "metadata": {}, "source": [ "Although not very useful, we can also generate and visualize a graph from a `delayed` function:" ] }, { "cell_type": "code", "execution_count": 10, "id": "59382145b51d5cf5", "metadata": { "ExecuteTime": { "end_time": "2025-04-14T14:58:59.293852Z", "start_time": "2025-04-14T14:58:58.906322Z" } }, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "G\n", "\n", "\n", "\n", "get_total_amount_sum\n", "\n", "file_url\n", "\n", "unit\n", "\n", "get_total_amount_sum\n", "\n", "result\n", "\n", "\n", "\n", "_e073c363db07469d8daa5887f9a5e42d\n", "\n", "result\n", "\n", "\n", "\n", "get_total_amount_sum:outputs_result->_e073c363db07469d8daa5887f9a5e42d\n", "\n", "\n", "\n", "\n", "\n", "input_file_url\n", "\n", "file_url\n", "\n", "\n", "\n", "input_file_url->get_total_amount_sum:inputs_file_url\n", "\n", "\n", "\n", "\n", "\n", "input_unit\n", "\n", "unit\n", "\n", "\n", "\n", "input_unit->get_total_amount_sum:inputs_unit\n", "\n", "\n", "\n", "\n", "" ], "text/plain": [ "" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "get_graph_image(get_total_amount_sum.to_graph().to_dot())" ] }, { "cell_type": "markdown", "id": "3f63e8d97c9e83d4", "metadata": {}, "source": [ "## Nested graphs\n", "\n", "What if we want to compose multiple graphs into one bigger graph? We can do that by nesting graph functions!" ] }, { "cell_type": "code", "execution_count": 11, "id": "63552d185b7bc52b", "metadata": { "ExecuteTime": { "end_time": "2025-04-14T14:58:59.312442Z", "start_time": "2025-04-14T14:58:59.308586Z" } }, "outputs": [], "source": [ "URL = \"https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_{year}-{month:02}.parquet\"\n", "\n", "\n", "@graph\n", "def nyc_yellow_taxi_fare_total_by_year(years, unit):\n", " \"\"\"\n", " Calculate the total fare amount for multiple years\n", "\n", " :param years: A list of years for which to calculate the total fare amount\n", " :param unit: The unit to divide the total amount sum by\n", " :return: Total fare amounts for the given years\n", " \"\"\"\n", " results = []\n", "\n", " for year in years:\n", " results.append(year)\n", " results.append(nyc_yellow_taxi_fare_total_for_year(year, unit))\n", "\n", " return collect_result(*results)" ] }, { "cell_type": "markdown", "id": "e89cbd56ea834894", "metadata": {}, "source": [ "Notice that when we visualize the graph that the `nyc_yellow_taxi_fare_total_for_year` function is now a node in the graph:" ] }, { "cell_type": "code", "execution_count": 12, "id": "89296f8561deb765", "metadata": { "ExecuteTime": { "end_time": "2025-04-14T14:58:59.876123Z", "start_time": "2025-04-14T14:58:59.318098Z" } }, "outputs": [ { "data": { "image/svg+xml": [ "\n", "\n", "G\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_8c48db206cd244efaa10be4bd351e580\n", "\n", "unit\n", "\n", "nyc_yellow_taxi_fare_total_for_year\n", "\n", "result\n", "\n", "\n", "\n", "collect_result_8240599c8d5e45f683995e6217a1f6dd\n", "\n", "results\n", "\n", "collect_result\n", "\n", "result\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_8c48db206cd244efaa10be4bd351e580:outputs_result->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_1\n", "\n", "\n", "\n", "\n", "\n", "input_unit\n", "\n", "unit\n", "\n", "\n", "\n", "input_unit->nyc_yellow_taxi_fare_total_for_year_8c48db206cd244efaa10be4bd351e580:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_f19c85dd4fe943daab94a44c7a6d364d\n", "\n", "unit\n", "\n", "nyc_yellow_taxi_fare_total_for_year\n", "\n", "result\n", "\n", "\n", "\n", "input_unit->nyc_yellow_taxi_fare_total_for_year_f19c85dd4fe943daab94a44c7a6d364d:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_4f3e2d249e5f45d3a6c6c43d96927c26\n", "\n", "unit\n", "\n", "nyc_yellow_taxi_fare_total_for_year\n", "\n", "result\n", "\n", "\n", "\n", "input_unit->nyc_yellow_taxi_fare_total_for_year_4f3e2d249e5f45d3a6c6c43d96927c26:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_8fe11d399a064cf8a175f7196f82acdb\n", "\n", "unit\n", "\n", "nyc_yellow_taxi_fare_total_for_year\n", "\n", "result\n", "\n", "\n", "\n", "input_unit->nyc_yellow_taxi_fare_total_for_year_8fe11d399a064cf8a175f7196f82acdb:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_33c25a8b8f24490b9c22eb2e6ca33313\n", "\n", "unit\n", "\n", "nyc_yellow_taxi_fare_total_for_year\n", "\n", "result\n", "\n", "\n", "\n", "input_unit->nyc_yellow_taxi_fare_total_for_year_33c25a8b8f24490b9c22eb2e6ca33313:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_6fb0ad4e8e214b978ebee0c1ddb6df19\n", "\n", "unit\n", "\n", "nyc_yellow_taxi_fare_total_for_year\n", "\n", "result\n", "\n", "\n", "\n", "input_unit->nyc_yellow_taxi_fare_total_for_year_6fb0ad4e8e214b978ebee0c1ddb6df19:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_06c7b8a6b7ee49da92c8086377e6b210\n", "\n", "unit\n", "\n", "nyc_yellow_taxi_fare_total_for_year\n", "\n", "result\n", "\n", "\n", "\n", "input_unit->nyc_yellow_taxi_fare_total_for_year_06c7b8a6b7ee49da92c8086377e6b210:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_91ed80f9e70b4362a74e1f90813b872f\n", "\n", "unit\n", "\n", "nyc_yellow_taxi_fare_total_for_year\n", "\n", "result\n", "\n", "\n", "\n", "input_unit->nyc_yellow_taxi_fare_total_for_year_91ed80f9e70b4362a74e1f90813b872f:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_111cb81888d848baa504eb0f3228097e\n", "\n", "unit\n", "\n", "nyc_yellow_taxi_fare_total_for_year\n", "\n", "result\n", "\n", "\n", "\n", "input_unit->nyc_yellow_taxi_fare_total_for_year_111cb81888d848baa504eb0f3228097e:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_1f0d6a5e35f340f5ac022bff3a4746b3\n", "\n", "unit\n", "\n", "nyc_yellow_taxi_fare_total_for_year\n", "\n", "result\n", "\n", "\n", "\n", "input_unit->nyc_yellow_taxi_fare_total_for_year_1f0d6a5e35f340f5ac022bff3a4746b3:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_2209041b2f064a919e3aafe1a128958e\n", "\n", "unit\n", "\n", "nyc_yellow_taxi_fare_total_for_year\n", "\n", "result\n", "\n", "\n", "\n", "input_unit->nyc_yellow_taxi_fare_total_for_year_2209041b2f064a919e3aafe1a128958e:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_aca165bfd37a47be8a3a974965d2e9ed\n", "\n", "unit\n", "\n", "nyc_yellow_taxi_fare_total_for_year\n", "\n", "result\n", "\n", "\n", "\n", "input_unit->nyc_yellow_taxi_fare_total_for_year_aca165bfd37a47be8a3a974965d2e9ed:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_2cee81f907694caaa44783c615184d4f\n", "\n", "unit\n", "\n", "nyc_yellow_taxi_fare_total_for_year\n", "\n", "result\n", "\n", "\n", "\n", "input_unit->nyc_yellow_taxi_fare_total_for_year_2cee81f907694caaa44783c615184d4f:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_ba8db8e98a6d4760a7ec8f4d9cc9ea43\n", "\n", "unit\n", "\n", "nyc_yellow_taxi_fare_total_for_year\n", "\n", "result\n", "\n", "\n", "\n", "input_unit->nyc_yellow_taxi_fare_total_for_year_ba8db8e98a6d4760a7ec8f4d9cc9ea43:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_a061db2b922447a6923a0e80de037ba4\n", "\n", "unit\n", "\n", "nyc_yellow_taxi_fare_total_for_year\n", "\n", "result\n", "\n", "\n", "\n", "input_unit->nyc_yellow_taxi_fare_total_for_year_a061db2b922447a6923a0e80de037ba4:inputs_unit\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_f19c85dd4fe943daab94a44c7a6d364d:outputs_result->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_3\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_4f3e2d249e5f45d3a6c6c43d96927c26:outputs_result->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_5\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_8fe11d399a064cf8a175f7196f82acdb:outputs_result->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_7\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_33c25a8b8f24490b9c22eb2e6ca33313:outputs_result->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_9\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_6fb0ad4e8e214b978ebee0c1ddb6df19:outputs_result->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_11\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_06c7b8a6b7ee49da92c8086377e6b210:outputs_result->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_13\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_91ed80f9e70b4362a74e1f90813b872f:outputs_result->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_15\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_111cb81888d848baa504eb0f3228097e:outputs_result->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_17\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_1f0d6a5e35f340f5ac022bff3a4746b3:outputs_result->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_19\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_2209041b2f064a919e3aafe1a128958e:outputs_result->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_21\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_aca165bfd37a47be8a3a974965d2e9ed:outputs_result->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_23\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_2cee81f907694caaa44783c615184d4f:outputs_result->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_25\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_ba8db8e98a6d4760a7ec8f4d9cc9ea43:outputs_result->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_27\n", "\n", "\n", "\n", "\n", "\n", "nyc_yellow_taxi_fare_total_for_year_a061db2b922447a6923a0e80de037ba4:outputs_result->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_29\n", "\n", "\n", "\n", "\n", "\n", "_27c3bfa7a0a2458b80ea0399bf0f32fa\n", "\n", "result\n", "\n", "\n", "\n", "collect_result_8240599c8d5e45f683995e6217a1f6dd:outputs_result->_27c3bfa7a0a2458b80ea0399bf0f32fa\n", "\n", "\n", "\n", "\n", "\n", "const__09eb8df583814d1482fa8e8d93c8793b\n", "\n", "2010\n", "\n", "\n", "\n", "const__09eb8df583814d1482fa8e8d93c8793b->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_0\n", "\n", "\n", "\n", "\n", "\n", "const__dca7765823634d62b11bd7cc932decc1\n", "\n", "2011\n", "\n", "\n", "\n", "const__dca7765823634d62b11bd7cc932decc1->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_2\n", "\n", "\n", "\n", "\n", "\n", "const__d91bb7be0f954c4da1446dc7cbe5f89c\n", "\n", "2012\n", "\n", "\n", "\n", "const__d91bb7be0f954c4da1446dc7cbe5f89c->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_4\n", "\n", "\n", "\n", "\n", "\n", "const__dbea3cf128994ea5b049c43f72a1f847\n", "\n", "2013\n", "\n", "\n", "\n", "const__dbea3cf128994ea5b049c43f72a1f847->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_6\n", "\n", "\n", "\n", "\n", "\n", "const__eef7d01226e442848a0ea1a914f2b736\n", "\n", "2014\n", "\n", "\n", "\n", "const__eef7d01226e442848a0ea1a914f2b736->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_8\n", "\n", "\n", "\n", "\n", "\n", "const__b4d54efc6e0b4c369aefc5b892e1285f\n", "\n", "2015\n", "\n", "\n", "\n", "const__b4d54efc6e0b4c369aefc5b892e1285f->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_10\n", "\n", "\n", "\n", "\n", "\n", "const__2c6144f926264c4eaa5d310d921f5438\n", "\n", "2016\n", "\n", "\n", "\n", "const__2c6144f926264c4eaa5d310d921f5438->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_12\n", "\n", "\n", "\n", "\n", "\n", "const__86131dc187db43b69b5b127d8d96a9d3\n", "\n", "2017\n", "\n", "\n", "\n", "const__86131dc187db43b69b5b127d8d96a9d3->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_14\n", "\n", "\n", "\n", "\n", "\n", "const__ce7b8fe18e954b4ebba7ab1871706e5a\n", "\n", "2018\n", "\n", "\n", "\n", "const__ce7b8fe18e954b4ebba7ab1871706e5a->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_16\n", "\n", "\n", "\n", "\n", "\n", "const__074af82f4e90435aa35d4343b7eb8b1c\n", "\n", "2019\n", "\n", "\n", "\n", "const__074af82f4e90435aa35d4343b7eb8b1c->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_18\n", "\n", "\n", "\n", "\n", "\n", "const__68ebe621f4614c759724696ac1ce8a30\n", "\n", "2020\n", "\n", "\n", "\n", "const__68ebe621f4614c759724696ac1ce8a30->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_20\n", "\n", "\n", "\n", "\n", "\n", "const__896c58a9347547409289a25e5cdd555d\n", "\n", "2021\n", "\n", "\n", "\n", "const__896c58a9347547409289a25e5cdd555d->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_22\n", "\n", "\n", "\n", "\n", "\n", "const__446c367a6d2641bc8f26cdf697c29485\n", "\n", "2022\n", "\n", "\n", "\n", "const__446c367a6d2641bc8f26cdf697c29485->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_24\n", "\n", "\n", "\n", "\n", "\n", "const__423db51e475741afbbcad946eafa0f2d\n", "\n", "2023\n", "\n", "\n", "\n", "const__423db51e475741afbbcad946eafa0f2d->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_26\n", "\n", "\n", "\n", "\n", "\n", "const__122a797dddbd474ab5f9d2c7919b9728\n", "\n", "2024\n", "\n", "\n", "\n", "const__122a797dddbd474ab5f9d2c7919b9728->collect_result_8240599c8d5e45f683995e6217a1f6dd:inputs_28\n", "\n", "\n", "\n", "\n", "" ], "text/plain": [ "" ] }, "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ "get_graph_image(\n", " nyc_yellow_taxi_fare_total_by_year.to_graph(years=range(2010, 2025)).to_dot()\n", ")" ] }, { "cell_type": "markdown", "id": "bad2b037c716ffa0", "metadata": {}, "source": [ "## Executing graphs\n", "\n", "First, we need to generate the task graph:\n", "\n" ] }, { "cell_type": "code", "execution_count": 13, "id": "e045ab2878133015", "metadata": { "ExecuteTime": { "end_time": "2025-04-14T14:58:59.886230Z", "start_time": "2025-04-14T14:58:59.881259Z" } }, "outputs": [], "source": [ "task_graph, keys = nyc_yellow_taxi_fare_total_by_year.to_graph(\n", " years=range(2024, 2025)\n", ").to_dict(unit=1_000_000)" ] }, { "cell_type": "markdown", "id": "d78628f402322567", "metadata": {}, "source": [ "Then we can execute it using `pargraph.GraphEngine`:" ] }, { "cell_type": "code", "execution_count": 14, "id": "303a84db8ad1e522", "metadata": { "ExecuteTime": { "end_time": "2025-04-14T15:00:06.111526Z", "start_time": "2025-04-14T14:58:59.892551Z" } }, "outputs": [], "source": [ "from pargraph import GraphEngine\n", "\n", "graph_engine = GraphEngine()\n", "result = graph_engine.get(task_graph, keys)" ] }, { "cell_type": "markdown", "id": "93a7176bf11b9329", "metadata": {}, "source": [ "And the result is:" ] }, { "cell_type": "code", "execution_count": 15, "id": "edbf0b33bd117cd6", "metadata": { "ExecuteTime": { "end_time": "2025-04-14T15:00:06.171306Z", "start_time": "2025-04-14T15:00:06.165385Z" } }, "outputs": [ { "data": { "text/plain": [ "[2024 1145.869126\n", " dtype: float64]" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "result" ] }, { "cell_type": "markdown", "id": "63538e63b325ed9", "metadata": {}, "source": [ "The results are returned as a list according to the keys structure:" ] }, { "cell_type": "code", "execution_count": 16, "id": "914f8dabd39cf4db", "metadata": { "ExecuteTime": { "end_time": "2025-04-14T15:00:06.305573Z", "start_time": "2025-04-14T15:00:06.300627Z" } }, "outputs": [ { "data": { "text/plain": [ "['node_output_collect_result_559f87ac40c44e1d96c6c4677e9922d1']" ] }, "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ "keys" ] }, { "cell_type": "markdown", "id": "501e9778496f5916", "metadata": {}, "source": [ "Hence, we can extract our result by indexing the first element:" ] }, { "cell_type": "code", "execution_count": 17, "id": "e097ae10f050b113", "metadata": { "ExecuteTime": { "end_time": "2025-04-14T15:09:08.532513Z", "start_time": "2025-04-14T15:09:08.527775Z" } }, "outputs": [ { "data": { "text/plain": [ "2024 1145.869126\n", "dtype: float64" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "result[0]" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.6" } }, "nbformat": 4, "nbformat_minor": 5 }