Add components overview and pipeline sequence for Microsoft.Spark <-> Apache Spark integration (#1189)

Author: grazy27

docs/img/.diagrams-source/Spark-dotnet-integration-component-diagram.puml

@@ -0,0 +1,62 @@
+
+@startuml Spark-dotnet-integration-component-diagram
+!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Component.puml
+HIDE_STEREOTYPE()
+skinparam legend {
+    FontColor #Black
+}
+skinparam dpi 300
+
+
+title: Microsoft Spark Component Diagram
+
+AddComponentTag("ApacheSpark", $sprite="img:./spark-logo.png{scale=0.25}", $legendText="Apache Spark components")
+AddComponentTag("dotnet", $sprite="img:./dotnet-logo.png{scale=0.25}")
+AddComponentTag("package", $sprite="img:./nuget-logo.png{scale=0.1}", $bgColor="#c09fe0")
+AddComponentTag("scala", $sprite="img:./scala-logo.png{scale=0.2}", $bgColor="#c09fe0")
+AddComponentTag("inThisRepo", $bgColor="#c09fe0", $legendText="Components that are defined in this repository")
+
+System(SparkDriver, "Spark Driver", "Entire system. Entrypoint, driver spark process, cluster manager...", $tags="ApacheSpark")
+
+Boundary(SparkWorkerContainer, "Spark Worker", "Single instance of worker"){
+    Component(SparkWorker, "Spark Worker Process", "Java process", "Apache Spark worker process, responsible for handling requests from the driver.", $tags="ApacheSpark")
+    Component(DotnetWorker, "Microsoft.Spark.Worker.exe", ".NET process", ".NET executable present on worker nodes. Started with the first request from the worker, and continuously processes tasks. .NET **UDFs** are executed here.", $tags="inThisRepo+dotnet")
+}
+
+Container(SparkMoreWorkers, "Additional Spark Workers", "Multiple instances of Spark Worker", $tags="ApacheSpark")
+
+Rel_D(SparkDriver, SparkWorker, "Manages instance, sends tasks", "")
+Rel_D(SparkDriver, SparkMoreWorkers, "Sends tasks to additional workers", "","","", "#blue")
+BiRel_D(SparkWorker, DotnetWorker, "Creates instance and sends tasks", "Binary over socket")
+
+note right on link
+    From Spark's perspective, it communicates with the PySpark worker.
+    Instead of the path to the Python binary, the path to the .NET worker is provided.
+    This allows the same API interaction as with PySpark,
+    missing yet APIs can be added by contributors.
+end note
+
+SparkWorker -[dotted,#blue]right- SparkMoreWorkers: Multiple worker instances
+Lay_R(SparkWorker, SparkMoreWorkers)
+
+Boundary(UserApp, "User Application"){
+    Component(MainApp, "User .NET Application", ".NET executable dll", "Application intended to work with Spark. Contains all user-defined code for Spark: pipeline definition, UDFs, ML, streaming, etc.", $tags="dotnet")
+    Component(DotnetSparkPackage, "Microsoft.Spark", "Nuget package", "Communicates with the Microsoft Spark bridge. Contains wrappers over Spark Java objects and API definitions.", $tags="package+inThisRepo")
+    Rel(MainApp, DotnetSparkPackage, "Depends on")
+}
+
+Component(MicrosoftScalaBridge, "Spark <-> .NET Bridge", "microsoft-spark-xxx.jar", "Entry point for the user app. Started by Spark when spark-submit is invoked. Starts the .NET user app and bridges all API calls to Spark.", $tags="scala+inThisRepo")
+
+Rel_L(MicrosoftScalaBridge, SparkDriver, "Creates Spark objects and controls their lifecycle", "jar loaded to Spark context")
+BiRel_L(MicrosoftScalaBridge, UserApp, "Handles all Spark API calls and results retrieval", "Binary over sockets")
+
+Person(user, "User")
+Rel_R(user, SparkDriver, "Executes 'spark-submit microsoft-spark-xxx.jar'")
+
+
+legend right
+<#GhostWhite,#black>|        |=__Legend__|
+|<#c09fe0>   | Components that are defined within this repository |
+endlegend
+
+@enduml

docs/img/.diagrams-source/Spark-dotnet-sequence-diagram-simple.puml

@@ -0,0 +1,89 @@
+@startuml Spark-dotnet-sequence-diagram-simple
+
+title "Sequence Diagram for Processing Simple Pipeline with Spark .NET\nWithout UDFs or Data Retrieval"
+
+skinparam dpi 300
+skinparam BoxPadding 10
+
+actor "User" as user
+
+box "Master Node"
+participant "Spark: Master" as spark_master
+participant "JVM<->.NET Bridge" as bridge
+participant "MyProgram.exe:\nUser .NET App" as dotnet_master
+participant "Microsoft.Spark\n(NuGet Package)" as dotnet_nuget
+end box
+
+box "Worker Node\n(One of Many)"
+participant "Spark: Worker" as spark_worker
+participant "Microsoft.Spark.Worker" as dotnet_worker
+end box
+
+user -> spark_master: Executes\n**spark-submit** microsoft-spark-xx.jar\n--files MyUdfs.dll MyProgram.zip
+activate spark_master
+
+spark_master -> bridge: Load and start executing JAR
+activate bridge
+bridge -> dotnet_master: Start MyProgram
+deactivate bridge
+
+activate dotnet_master
+dotnet_master -> dotnet_nuget: Build SparkSession
+deactivate dotnet_master
+activate dotnet_nuget
+
+dotnet_nuget -> bridge: Connect to socket,\nRequest Spark Session creation
+activate bridge
+return Reference to JVM object SparkSession
+note over dotnet_nuget
+    Each .NET Spark-related object has a JvmObjectReference.
+    Whenever a method/property call on these objects is requested,
+    it is broadcasted over the socket to the bridge,
+    where actual execution occurs.
+end note
+return Session
+activate dotnet_master
+
+note over dotnet_master
+    Pipeline execution:
+    ""_spark""
+    ""    .Read()""
+    ""    .Parquet($"C:\data.parquet")""
+    ""    .GroupBy(Col("Faculty"))""
+    ""    .Agg(Avg(Col("Grage")))""
+    ""    .Write()""
+    ""    .Parquet(@"C:\averageGrades.parquet");""
+end note
+
+dotnet_master -> dotnet_nuget: Invocations on objects in Microsoft.Spark
+deactivate dotnet_master
+activate dotnet_nuget
+dotnet_nuget -> bridge: In binary, smth similar to: \n ""{ref:123, m: "GroupBy", args:[arg1, arg2]}"" \n \t\t\t\t\t•••••••• \n ""{ref:125, m: "Parquet", args:["path"]}""
+deactivate dotnet_nuget
+activate bridge
+
+bridge -> spark_master: Invocations on actual\nSpark objects
+deactivate bridge
+
+spark_master -> spark_master: Load data,\nGenerate execution graph,\nCreate RDD
+
+spark_master -> spark_worker: Create tasks for processing partitions of RDD in a distributed manner
+activate spark_worker
+return Processed result
+spark_master -> spark_master: Aggregate results from workers\nWrite to Parquet
+spark_master --> bridge
+activate bridge
+bridge --> dotnet_nuget
+deactivate bridge
+activate dotnet_nuget
+dotnet_nuget --> dotnet_master
+deactivate dotnet_nuget
+
+activate dotnet_master
+dotnet_master --> bridge: Execution complete
+deactivate dotnet_master
+activate bridge
+bridge --> spark_master: Execution complete
+deactivate bridge
+return Execution complete
+@enduml

docs/img/.diagrams-source/Spark-dotnet-sequence-diagram-udf-data.puml

@@ -0,0 +1,156 @@
+@startuml Spark-dotnet-sequence-diagram-udf-data
+title "Sequence Diagram for Processing Pipeline with Spark .NET: UDF & Data retrieval"
+
+skinparam dpi 200
+skinparam BoxPadding 10
+
+actor "User" as user
+
+box "Master Node"
+participant "Spark: Master" as spark_master
+participant "JVM<->.NET Bridge" as bridge
+participant "MyProgram.exe:\nUser .NET App" as dotnet_master
+participant "Microsoft.Spark\n(NuGet Package)" as dotnet_nuget
+end box
+
+box "Worker Node\n(One of Many)"
+participant "Spark: Worker" as spark_worker
+participant "Microsoft.Spark.Worker" as dotnet_worker
+end box
+
+user -> spark_master: Executes \n**spark-submit** microsoft-spark-xx.jar\n--files MyUdfs.dll MyProgram.zip
+activate spark_master
+
+spark_master -> bridge: Load and start executing jar
+activate bridge
+bridge -> dotnet_master: Start MyProgram
+deactivate bridge
+
+activate dotnet_master
+dotnet_master -> dotnet_nuget: Build SparkSession
+deactivate dotnet_master
+activate dotnet_nuget
+
+dotnet_nuget -> bridge: Connect to socket,\nRequest Spark Session creation
+activate bridge
+bridge -> spark_master: Request Spark Session creation
+return Reference to JVM object SparkSession
+return Session
+activate dotnet_master
+
+group "Register UDF"
+    note over dotnet_master
+        ""var df = LoadDataFromSomeWhere();"" // This part is ommitted
+        ""Func<Column, Column> udfArray =""
+            ""Udf<string, string[]>(str => [str, $"{str}-{str.Length}"]);""
+    end note
+
+    dotnet_master -> dotnet_nuget: Func<> object
+    deactivate dotnet_master
+    activate dotnet_nuget
+    dotnet_nuget -> dotnet_nuget: Serialize Func using binary serializer
+    dotnet_nuget -> bridge: Invoke UDF creation,\nPass serialized UDF as a parameter
+    deactivate dotnet_nuget
+    activate bridge
+    bridge -> spark_master: Register UDF as a PythonFunction,\nSpecify Microsoft.Spark.Worker.exe instead of Python.exe\nDeclare serialized UDF as an argument
+    deactivate bridge
+
+    spark_master -> spark_master: Register a Python UDF
+
+    spark_master --> bridge
+    activate bridge
+    bridge --> dotnet_nuget: UDF JVM reference
+    deactivate bridge
+
+    activate dotnet_nuget
+    dotnet_nuget --> dotnet_master
+    deactivate dotnet_nuget
+    activate dotnet_master
+end
+
+group "Invoke UDF"
+    note over dotnet_master
+        // Cache() needed for immediate invocation,
+        // otherwise df invoked lazily when needed
+        ""var arrayDF =""
+            ""df.Select(Explode(udfArray(df["value"])))""
+            "".Cache();""
+    end note
+
+
+    dotnet_master -> dotnet_nuget
+    deactivate dotnet_master
+    activate dotnet_nuget
+
+    dotnet_nuget -> bridge: Pass calls to bridge
+    deactivate dotnet_nuget
+    activate bridge
+    bridge -> spark_master: Load data,\nGenerate execution graph,\nCreate RDD
+    deactivate bridge
+
+    spark_master -> spark_worker: Create tasks for processing partitions of RDD
+    activate spark_worker
+    spark_worker -> dotnet_worker: Start process,\nInitiate socket connection,\nPass task content and serialized UDF
+    activate dotnet_worker
+
+    dotnet_worker -> dotnet_worker: Deserialize Func and execute it\nPass arguments received from Spark worker
+    return UDF execution result
+    return
+
+    spark_master -> spark_master: Aggregate results from workers
+    spark_master --> bridge
+    activate bridge
+    bridge --> dotnet_nuget
+    deactivate bridge
+    activate dotnet_nuget
+    dotnet_nuget --> dotnet_master
+    deactivate dotnet_nuget
+    activate dotnet_master
+end
+
+group "Fetch Dataset in .NET Memory"
+    note over dotnet_master
+        ""var result =""
+            ""arrayDF.Collect().ToList();""
+    end note
+
+    dotnet_master -> dotnet_nuget: Collect dataset
+    deactivate dotnet_master
+    activate dotnet_nuget
+    dotnet_nuget -> bridge: Request dataset collection
+    deactivate dotnet_nuget
+
+    activate bridge
+    bridge -> spark_master: .Collect() request
+
+    deactivate bridge
+
+    spark_master --> bridge: Collected data
+    activate bridge
+
+    bridge --> dotnet_nuget: Collected data
+    deactivate bridge
+    activate dotnet_nuget
+    dotnet_nuget -> bridge: Initiate broadcast of all rows via socket
+    deactivate dotnet_nuget
+    activate bridge
+
+    bridge -> dotnet_nuget: Entire dataset serialized in Python Pickle format\n**Expensive operation**
+
+    deactivate bridge
+
+    activate dotnet_nuget
+    dotnet_nuget --> dotnet_master: Deserialized row collection
+    deactivate dotnet_nuget
+    activate dotnet_master
+
+end
+
+activate dotnet_master
+dotnet_master --> bridge: Execution complete
+deactivate dotnet_master
+activate bridge
+bridge --> spark_master: Execution complete1
+deactivate bridge
+return Execution complete.
+@enduml

docs/img/.diagrams-source/dotnet-logo.png

@@ -0,0 +1,33 @@
+
+# Components Overview
+
+The following component diagram depicts a high-level overview of the vital components participating in a .NET for Apache Spark application lifecycle:
+
+![Icon](img/Spark-dotnet-integration-component-diagram.png)
+
+This diagram illustrates the interaction between the Apache Spark components and the .NET components. The `Microsoft.Spark` NuGet package contains a number of wrappers over the Scala JVM objects (with reference to internal objects mapped in the bridge), and allows calling various methods on those objects. Unless `Collect()` is called on a dataframe, no data is actually loaded into the .NET app; the solution acts as a proxy for the Scala Spark API. Key components include:
+
+- **Spark Driver**: The main entry point for Spark jobs, responsible for managing the job lifecycle and distributing tasks to worker nodes.
+- **Spark Worker**: Executes tasks sent by the driver, processing data and performing computations.
+- **Microsoft.Spark.Worker**: A .NET executable that runs on worker nodes, allowing .NET UDFs to be executed as part of the Spark job.
+- **User .NET Application**: Contains the user-defined code for interacting with Spark, including data processing pipelines, UDFs, and more.
+- **JVM<->.NET Bridge**: Facilitates communication between the .NET application and the JVM-based Spark components.
+
+# Pipeline Processing Sequence
+
+## Basic Sequence Diagram
+
+The basic sequence diagram for the application lifecycle is depicted below:
+
+![Icon](img/Spark-dotnet-sequence-diagram-simple.png)
+
+This diagram shows the flow of control and data during the execution of a simple Spark pipeline with .NET, without UDFs or data retrieval. Note that Worker is never instantiated, and no actual pipeline data is transferred to .NET.
+
+
+## Sequence Diagram with UDFs and Data Retrieval
+
+The sequence diagram below illustrates a more complex scenario involving user-defined functions (UDFs) and data retrieval:
+
+![Icon](img/Spark-dotnet-sequence-diagram-udf-data.png)
+
+This diagram includes the steps for registering and invoking UDFs, as well as fetching the dataset into .NET memory.