Improve READMEs consistency (#59)

* Updated READMEs for consistency
* Improve "Running in IntelliJ" section for Java examples
* Better clarify connecting to VPC-networked services

Author: nicusX

java/CustomMetrics/README.md

@@ -3,18 +3,32 @@
 * Flink version: 1.20
 * Flink API: DataStream API
 * Language: Java (11)
+* Flink connectors: Kinesis Sink
 
 This example demonstrates how to create your own metrics to track application-specific data, such as processing events or accessing external resources and publish it to Cloudwatch.
 
+The applications generates data internally and writes to a Kinesis Stream.
+
 ### Runtime configuration
 
-The application reads the runtime configuration from the Runtime Properties, when running on Amazon Managed Service for Apache Flink.
+When running on Amazon Managed Service for Apache Flink the runtime configuration is read from *Runtime Properties*.
+
+When running locally, the configuration is read from the [`resources/flink-application-properties-dev.json`](src/main/resources/flink-application-properties-dev.json) file located in the resources folder.
+
+Runtime parameters:
+
+| Group ID        | Key           | Description               | 
+|-----------------|---------------|---------------------------|
+| `OutputStream0` | `stream.name` | Name of the output stream |
+| `OutputStream0`  | `aws.region`  | (optional) Region of the output stream. If not specified, it will use the application region or the default region of the AWS profile, when running locally. |
+
+All parameters are case-sensitive.
+
+This simple example assumes the Kinesis Stream is in the same region as the application, or in the default region for the authentication profile, when running locally.
 
-Runtime Properties are expected in the Group ID `OutputStream0`. They are all case-sensitive:
-* `stream.name` - Kinesis Data Stream to be used for sink.
-* `aws.region` - AWS Region containing test resources.
 
 ### Running locally in IntelliJ
-Update `PropertyMap` in [configuration file](src/main/resources/flink-application-properties-dev.json).
 
-To start the Flink job in IntelliJ edit the Run/Debug configuration enabling *'Add dependencies with "provided" scope to the classpath'*.
+You can run this example directly in IntelliJ, without any local Flink cluster or local Flink installation.
+
+See [Running examples locally](../running-examples-locally.md) for details.

java/GettingStarted/README.md

@@ -5,32 +5,34 @@ Skeleton project for a basic Flink Java application to run on Amazon Managed Ser
 * Flink version: 1.20
 * Flink API: DataStream API
 * Language: Java (11)
+* Flink connectors: Kinesis Consumer, Kinesis Sink
 
 The project can run both on Amazon Managed Service for Apache Flink, and locally for development.
 
-The application shows how to get runtime configuration, and sets up a Kinesis Data Stream source and a sink.
+The application shows how to get runtime configuration, and set up a Kinesis Data Stream source and a sink.
 
 ### Runtime configuration
 
-The application reads the runtime configuration from the Runtime Properties, when running on Amazon Managed Service for 
-Apache Flink, or from command line parameters, when running locally.
+When running on Amazon Managed Service for Apache Flink the runtime configuration is read from *Runtime Properties*.
 
-Runtime Properties are expected in the Group ID `FlinkApplicationProperties`. 
-Command line parameters should be prepended by `--`.
+When running locally, the configuration is read from the [`resources/flink-application-properties-dev.json`](resources/flink-application-properties-dev.json) file located in the resources folder.
 
-They are all case-sensitive.
+Runtime parameters:
 
-Configuration parameters:
+| Group ID        | Key           | Description               | 
+|-----------------|---------------|---------------------------|
+| `InputStream0`  | `stream.name` | Name of the input stream  |
+| `InputStream0`  | `aws.region`  | (optional) Region of the input stream. If not specified, it will use the application region or the default region of the AWS profile, when running locally. |
+| `OutputStream0` | `stream.name` | Name of the output stream |
+| `OutputStream0`  | `aws.region`  | (optional) Region of the output stream. If not specified, it will use the application region or the default region of the AWS profile, when running locally. |
 
-* `InputStreamRegion` region of the input stream (default: `us-east-1`)
-* `InputStreamName` name of the input Kinesis Data Stream (default: `ExampleInputStream`)
-* `OutputStreamRegion` region of the input stream (default: `us-east-1`)
-* `OutputStreamName` name of the input Kinesis Data Stream (default: `ExampleOutputStream`)
+All parameters are case-sensitive.
 
 ### Running in IntelliJ
 
-To start the Flink job in IntelliJ edit the Run/Debug configuration enabling *'Add dependencies with "provided" scope to 
-the classpath'*.
+You can run this example directly in IntelliJ, without any local Flink cluster or local Flink installation.
+
+See [Running examples locally](../running-examples-locally.md) for details.
 
 ### Generating data
 

java/GettingStartedTable/README.md

@@ -5,6 +5,7 @@ Example of project for a basic Flink Java application using the Table API & SQL
 * Flink version: 1.20
 * Flink API: Table API & SQL, and DataStream API
 * Language: Java (11)
+* Flink connectors: Kinesis Sink
 
 The project can run both on Amazon Managed Service for Apache Flink, and locally for development.
 
@@ -16,20 +17,21 @@ control of the generated data. In this example we implemented a `GeneratorFuncti
 
 ### Runtime configuration
 
-The application reads the runtime configuration from the Runtime Properties, when running on Amazon Managed Service for
-Apache Flink, or from command line parameters, when running locally.
+When running on Amazon Managed Service for Apache Flink the runtime configuration is read from *Runtime Properties*.
 
-Runtime Properties are expected in the Group ID `FlinkApplicationProperties`.
+When running locally, the configuration is read from the [`resources/flink-application-properties-dev.json`](resources/flink-application-properties-dev.json) file located in the resources folder.
 
-Command line parameters should be prepended by `--`.
+Runtime parameters:
 
-Configuration parameters:
+| Group ID        | Key           | Description               | 
+|-----------------|---------------|---------------------------|
+| `bucket`        | `name`        | Name of the destination bucket, **without** the prefix "s3://" |
+| `bucket`        | `path`        | Path withing the bucket the output will be written to, without the trailing "/" |
 
-* `s3Path` <s3-bucket>/<path> of the S3 destination , **without** the prefix `s3://`
-
-They parameters all case-sensitive.
+All parameters are case-sensitive.
 
 ### Running in IntelliJ
 
-To start the Flink job in IntelliJ edit the Run/Debug configuration enabling *'Add dependencies with "provided" scope to
-the classpath'*.
+You can run this example directly in IntelliJ, without any local Flink cluster or local Flink installation.
+
+See [Running examples locally](../running-examples-locally.md) for details.

java/KafkaConfigProviders/Kafka-SASL_SSL-ConfigProviders/README.md

@@ -160,7 +160,11 @@ is controlled via networking (SecurityGroups, NACL) and by the SASL credentials
 
 ### Application configuration parameters
 
-The application expects the following Runtime properties:
+When running on Amazon Managed Service for Apache Flink the runtime configuration is read from *Runtime Properties*.
+
+When running locally, the configuration is read from the [`resources/flink-application-properties-dev.json`](resources/flink-application-properties-dev.json) file located in the resources folder.
+
+Runtime parameters:
 
 | Group ID  | Key                                 | Description                                                                                            | 
 |-----------|-------------------------------------|--------------------------------------------------------------------------------------------------------|
@@ -172,3 +176,14 @@ The application expects the following Runtime properties:
 | `Output0` | `credentials.secret`                | Name of the secret (not the ARN) in SecretsManager containing the SASL/SCRAM credentials               |
 | `Output0` | `credentials.secret.username.field` | Name of the field (the key) of the secret, containing the SASL username. Optional, default: `username` |
 | `Output0` | `credentials.secret.password.field` | Name of the field (the key) of the secret, containing the SASL password. Optional, default: `password` |
+
+All parameters are case-sensitive.
+
+## Running locally in IntelliJ
+
+> Due to MSK VPC networking, to run this example on your machine you need to set up network connectivity to the VPC where MSK is deployed, for example with a VPN.
+> Setting this connectivity depends on your set up and is out of scope for this example.
+
+You can run this example directly in IntelliJ, without any local Flink cluster or local Flink installation.
+
+See [Running examples locally](../running-examples-locally.md) for details.

java/KafkaConfigProviders/Kafka-mTLS-Keystore-ConfigProviders/README.md

@@ -3,7 +3,7 @@
 * Flink version: 1.19
 * Flink API: DataStream API
 * Language: Java (11)
-* Connectors: Kafka (mTLS authentication)
+* Flink connectors: Kafka (mTLS authentication)
 
 This sample illustrates how to configure the Flink Kafka connectors (KafkaSource and KafkaSink) 
 retrieving custom KeyStore and TrustStore at runtime, using Config Providers.
@@ -71,10 +71,11 @@ Access Policy/Role associated with the application that is running a config prov
 
 ### Runtime configuration
 
-The application reads the runtime configuration from the Runtime Properties, when running on Amazon Managed Service for Apache Flink,
-or from `flink-application-properties-dev.json`, when running locally.
+When running on Amazon Managed Service for Apache Flink the runtime configuration is read from *Runtime Properties*.
 
-Runtime Properties are expected in the Group ID `Input0` and they are all case-sensitive.
+When running locally, the configuration is read from the [`resources/flink-application-properties-dev.json`](resources/flink-application-properties-dev.json) file located in the resources folder.
+
+Runtime parameters:
 
 | GroupId | Key                     | Default     | Description                                                        |
 |---------|-------------------------|-------------|--------------------------------------------------------------------|
@@ -89,10 +90,13 @@ Runtime Properties are expected in the Group ID `Input0` and they are all case-s
 | `Input0` | `keystore.secret`       |             | SecretManager secret ID  containing the password of the keystore   |
 | `Input0` | `keystore.secret.field` |             | SecretManager secret key containing the password of the keystore   |
 
+All parameters are case-sensitive.
 
 ## Running locally in IntelliJ
 
-To run the application in IntelliJ
+> Due to MSK VPC networking, to run this example on your machine you need to set up network connectivity to the VPC where MSK is deployed, for example with a VPN.
+> Setting this connectivity depends on your set up and is out of scope for this example.
+
+You can run this example directly in IntelliJ, without any local Flink cluster or local Flink installation.
 
-1. Edit the Run/Debug configuration enabling *'Add dependencies with "provided" scope to the classpath'*
-2. Update `flink-application-properties-dev.json` with property values (`bootstrap.servers`, `topic` etc.) that fit your environment.
+See [Running examples locally](../running-examples-locally.md) for details.

java/KafkaConnectors/README.md

@@ -9,35 +9,40 @@ This example demonstrate how to use
 [Flink Kafka Connector](https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/datastream/kafka/),
 source and sink.
 
-This example uses `KafkaSource` and `KafkaSink`.
+This example uses KafkaSource and KafkaSink.
 
 ![Flink Example](images/flink-example.png),
 
+> In this example, the Kafka Sink uses *exactly-once* delivery guarantees. This leverages Kafka transaction under the hood, improving guarantees but 
+> adding some overhead and increasing the effective latency of the output to the consumers of the destination Kafka topic. 
+> 
+> Moreover, there are failure scenarios were the Kafka Sink may still cause duplicates, even when set for exactly-once guarantees.
+> See [FLIP-319](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255071710) for more details.
+>
+> We recommend not to consider Kafka Sink *exactly-once* guarantees as a default setting for all sinks to Kafka. 
+> Make sure you understand the implications before enabling it. Refer to the [Flink Kafka sink documentation](https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/connectors/datastream/kafka/#fault-tolerance) for details.
+
 Note that the old 
 [`FlinkKafkaConsumer`](https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/datastream/kafka/#kafka-sourcefunction)
 and [`FlinkKafkaProducer`](https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/datastream/kafka/#kafka-producer)
 were removed in Flink 1.17 and 1.15, respectively.
 
 ## Runtime configuration
 
-The application reads the runtime configuration from the Runtime Properties, when running on Amazon Managed Service for Apache Flink,
-or from command line parameters, when running locally.
-
-Runtime Properties are expected in the Group IDs `Input0` and `Output0` (see [`resources/flink-application-properties-dev.json`](resources/flink-application-properties-dev.json)).
-
-All properties are case-sensitive.
+When running on Amazon Managed Service for Apache Flink the runtime configuration is read from *Runtime Properties*.
 
-Configuration parameters:
+When running locally, the configuration is read from the [`resources/flink-application-properties-dev.json`](resources/flink-application-properties-dev.json) file located in the resources folder.
 
-For the source (i.e. Group ID `Input0`):
-* `bootstrap.servers` source cluster boostrap servers
-* `topic` source topic (default: `source`)
-* `group.id` source group id (default: `my-group`)
+Runtime parameters:
 
-For the sink (i.e. Group ID `Output0`):
-* `bootstrap.servers` sink cluster bootstrap servers
-* `topic` sink topic (default: `destination`)
-* `transaction.timeout.ms` sink transaction timeout (default: `1000`)
+| Group ID  | Key                 | Description                       | 
+|-----------|---------------------|-----------------------------------|
+| `Input0`  | `bootstrap.servers` | Source cluster boostrap servers.  |
+| `Input0`  | `topic`             | Source topic (default: `source`). |
+| `Input0`  | `group.id`          | Source group id (default: `my-group`) |
+| `Output0` | `bootstrap.servers` | Destination cluster bootstrap servers. |
+| `Output0` | `topic`             | Destination topic (default: `destination`). |
+| `Output0` | `transaction.timeout.ms` | Sink transaction timeout (default: `1000`) |
 
 If you are connecting with no-auth and no SSL, above will work. Else you need additional configuration for both source and sink.
 
@@ -54,9 +59,10 @@ When using IAM Auth, the following Runtime Properties are expected at the Group
 
 ## Running locally in IntelliJ
 
-To run this example locally - 
-* Run a Kafka cluster locally. You can refer https://kafka.apache.org/quickstart to download and start Kafka locally.
-* Create `source` and `sink` topics. 
-* To start the Flink job in IntelliJ edit the Run/Debug configuration enabling *'Add dependencies with "provided" scope to the classpath'*.
-* Update [`resources/flink-application-properties-dev.json`](resources/flink-application-properties-dev.json)
-* Execute using credentials with permissions to consume data from a Kinesis Data Stream and write data into Amazon S3.
+> Due to MSK VPC networking, to run this example on your machine you need to set up network connectivity to the VPC where MSK is deployed, for example with a VPN.
+> Alternatively, you can use a local Kafka installation, for example in a container.
+> Setting up the connectivity or a local containerized Kafka depends on your set up and is out of scope for this example.
+
+You can run this example directly in IntelliJ, without any local Flink cluster or local Flink installation.
+
+See [Running examples locally](../running-examples-locally.md) for details.

java/KafkaConnectors/src/main/resources/flink-application-properties-dev.json

@@ -4,7 +4,7 @@
     "PropertyMap": {
       "bootstrap.servers": "<BootstrapServers>",
       "topic": "<Topic>",
-      "group.id": "<S3Path>"
+      "group.id": "<ConsumerGroupID>"
     }
   },
   {