Session + Spark example #1022
Open
Session + Spark example #1022
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for neo4j-graph-data-science-client ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
DarthMax
force-pushed
the
sessions_expose_arrow_client
branch
from
e511a66 to
95a787a
Compare
Co-authored-by: Florentin Dörre <florentin.dorre@neo4j.com>
Contributor
|
Closes #991 |
FlorentinD
reviewed
Dec 1, 2025
| "# Create a GDS session!\n", | ||
| "gds = sessions.get_or_create(\n", | ||
| " # we give it a representative name\n", | ||
| " session_name=\"bike_trips\",\n", |
Contributor
There was a problem hiding this comment.
i read about bike trips just today in our logs. nice to learn its your workload 👀
Mats-SX
requested changes
Dec 8, 2025
Contributor
Mats-SX
left a comment
Mats-SX
left a comment
There was a problem hiding this comment.
I have many textual changes on the notebook.
I think this is a great step forward! I think Nathan and Stu will be very happy to see it.
| return None | ||
|
|
||
| def aborted(self) -> bool: | ||
| return self.status == "Aborted" |
Contributor
There was a problem hiding this comment.
for consistency, let's also do the .lower() on this one
| "Flight returned internal error, with message: org.apache.arrow.flight.FlightRuntimeException: ", "" | ||
| ) | ||
| improved_message = improved_message.replace( | ||
| "Failed to invoke procedure `gds.arrow.project`: Caused by: org.apache.arrow.flight.FlightRuntimeException: ", |
Contributor
There was a problem hiding this comment.
this procedure doesn't really exist anymore -- only in very old Neo4j versions. these days it's called gds.arrow.project.v2 or, more commonly v3
you just moved it, I realise, but I want to note the ineffectiveness here.
|
|
||
| class CatalogEndpoints(ABC): | ||
| @abstractmethod | ||
| def get(self, graph_name: str) -> GraphV2: |
Contributor
There was a problem hiding this comment.
what if it doesn't find the graph? error?
|
|
||
| def get(self, graph_name: str) -> GraphV2: | ||
| if not self.list(graph_name): | ||
| raise ValueError(f"A graph with name '{graph_name}' does not exist in the catalog.") |
| from pandas import DataFrame | ||
|
|
||
| from graphdatascience.arrow_client.v1.gds_arrow_client import GdsArrowClient | ||
| from graphdatascience.arrow_client.v2.gds_arrow_client import GdsArrowClient |
Contributor
There was a problem hiding this comment.
one usage of this class was removed with another import. is this import now unused?
Contributor
There was a problem hiding this comment.
ah no it is used in the create factory
| "The notebook shows how to use the `graphdatascience` Python library to create, manage, and use a GDS Session.\n", | ||
| "\n", | ||
| "We consider a graph of bicycle rentals, which we're using as a simple example to show how project data from Spark to a GDS Session, run algorithms, and eventually retrieving the results back to Spark.\n", | ||
| "We will cover all management operations: creation, listing, and deletion." |
Contributor
There was a problem hiding this comment.
Suggested change
| "We will cover all management operations: creation, listing, and deletion." | |
| "In this notebook we will focus on the interaction with Apache Spark, and will not cover all possible actions using GDS sessions. We refer to other Tutorials for additional details." |
| "\n", | ||
| "Once the computation is done, we might want to further use the result in Spark.\n", | ||
| "We can do this in a similar way to the projection, by streaming batches of data into each of the Spark workers.\n", | ||
| "Retrieving the data is a bit more complicated since we need some input data frame in order to trigger computations on the Spark workers.\n", |
Contributor
There was a problem hiding this comment.
Suggested change
| "Retrieving the data is a bit more complicated since we need some input data frame in order to trigger computations on the Spark workers.\n", | |
| "Retrieving the data is a bit more complicated since we need some input DataFrame in order to trigger computations on the Spark workers.\n", |
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# 1. Start the node property export on the session\n", |
Contributor
There was a problem hiding this comment.
Suggested change
| "# 1. Start the node property export on the session\n", | |
| "# 1. Start the node property export on the GDS session\n", |
| "source": [ | ||
| "## Cleanup\n", | ||
| "\n", | ||
| "Now that we have finished our analysis, we can delete the session and stop the spark connection.\n", |
Contributor
There was a problem hiding this comment.
Suggested change
| "Now that we have finished our analysis, we can delete the session and stop the spark connection.\n", | |
| "Now that we have finished our analysis, we can delete the GDS session and stop the Spark session.\n", |
| "\n", | ||
| "Now that we have finished our analysis, we can delete the session and stop the spark connection.\n", | ||
| "\n", | ||
| "Deleting the session will release all resources associated with it, and stop incurring costs." |
Contributor
There was a problem hiding this comment.
Suggested change
| "Deleting the session will release all resources associated with it, and stop incurring costs." | |
| "Deleting the GDS session will release all resources associated with it, and stop incurring costs." |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ref GDSA-373
ref GDSA-469