Merge branch 'main' into main

Author: jeremyber-aws

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,30 @@
+<!--
+*Thank you for contributing to the Amazon Managed Service for Apache Flink examples - we are happy that you want to help us improve our examples. Please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
+
+## Contribution Checklist
+
+- Verify your PR follows the contribution recommendation of the repository
+- Each PR should add/modify a single example and address a single concern
+- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
+-->
+
+## Purpose of the change
+
+*For example: modify the Java Kinesis Sink to provide the stream ARN*
+
+## Verifying this change
+
+Please test your changes both running locally, in the IDE, and in Managed Service for Apache Flink. All examples must run
+in both environment without code changes.
+
+Describe how you tested your application, show the output of the running application with screenshots.
+
+## Significant changes
+
+*(Please check any boxes [x] if the answer is "yes". You can first publish the PR and check them afterward, for convenience.)*
+
+- [ ] Completely new example
+- [ ] Updated an existing example to newer Flink version or dependencies versions
+- [ ] Improved an existing example
+- [ ] Modified the runtime configuration of an existing example (i.e. added/removed/modified any runtime properties)
+- [ ] Modified the expected input or output of an existing example (e.g. modified the source or sink, modified the record schema)

.gitignore

@@ -12,4 +12,4 @@ env/
 venv/
 .java-version
 /pyflink/
-*-dev.json*
+

CONTRIBUTING.md

@@ -6,6 +6,70 @@ documentation, we greatly value feedback and contributions from our community.
 Please read through this document before submitting any issues or pull requests to ensure we have all the necessary
 information to effectively respond to your bug report or contribution.
 
+## Goal of this repository
+
+The goal of this repository is to provide working example of common patterns for Apache Flink in general, and, in 
+particular, for [Amazon Managed Service for Apache Flink](https://aws.amazon.com/managed-service-apache-flink).
+Each example illustrates a single, specific pattern, trying to make it simple removing external dependencies when possible.
+
+The examples also try to illustrate best practices and consistent approaches to common problems.
+
+The goal of this repository is not to illustrate *solutions*, end-to-end architectures.
+
+The AWS team managing the repository reserves the right to modify or reject new example proposals.
+
+### Guidelines or new examples
+
+* An example should focus on a single Flink API, unless the goal is to show the integration between different APIs.
+* Unless the goal of the example is showing a particular sink, the application should sink data to Kinesis Data Stream,
+    as JSON. This makes the output more easily verifiable, simplify the setup when running locally, and testing the application.
+* Examples should all work both locally, in the IDE (e.g. IntelliJ, PyCharm), and on Managed Service for Apache Flink,
+  without any code changes.
+
+#### Runtime configuration
+
+* follow the pattern used in
+  [this example](https://github.com/aws-samples/amazon-managed-service-for-apache-flink-examples/tree/main/java/GettingStarted)
+  to provide the configuration when running locally.
+* Use separate `PropertyGroupID` for each component, for example for each source and sink. Name them groups and  
+  properties consistently. See at
+  [this configuration](https://github.com/aws-samples/amazon-managed-service-for-apache-flink-examples/blob/main/java/Windowing/src/main/resources/flink-application-properties-dev.json)
+  as an example.
+  * The committed `flink-application-properties-dev.json` file MUST NOT contain any real resource ARN, Account ID, URL, 
+  or any secrets. Use placeholders instead (e.g. `arn:aws:kinesis:<region>:<accountId>:stream/OutputStream`) or obviously
+  fake resource names. 
+ 
+#### External resources
+
+* Minimize the external resources.
+* If the application requires external resources, provide instructions in the README.
+* Providing an additional CloudFormation template is optional and does not exempt from human-readable instructions in 
+  the README.
+
+#### External data generators
+
+* Avoid depending on external data generators as possible. Use the [DataGen connector](https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/connectors/datastream/datagen/)
+  to generate synthetic data.
+* If the example requires an external data generator, try to use [this one](https://github.com/aws-samples/amazon-managed-service-for-apache-flink-examples/tree/main/python/data-generator), 
+  or include it in the example. 
+
+#### README and documentation
+
+* Write an exhaustive README explaining what is the goal of the example, how the application works, Flink and connector
+  versions, external dependencies, permissions, and runtime configuration. Use [this example](https://github.com/aws-samples/amazon-managed-service-for-apache-flink-examples/tree/main/java/KafkaConfigProviders/Kafka-SASL_SSL-ConfigProviders)
+  as a reference.
+* Make sure the example works with what explained in the README, and without any implicit dependency or configuration.
+
+#### AWS authentication and credentials
+
+* AWS credentials must never be explicitly passed to the application. 
+* Any permissions must be provided from the IAM Role assigned to the Managed Apache Flink application. When running locally, leverage the IDE AWS plugins.
+
+#### Dependencies in PyFlink examples
+  * Use the pattern illustrated by [this example](https://github.com/aws-samples/amazon-managed-service-for-apache-flink-examples/tree/main/python/GettingStarted)
+    to provide JAR dependencies and build the ZIP using Maven.
+  * If the application also requires Python dependencies, use the pattern illustrated by [this example](https://github.com/aws-samples/amazon-managed-service-for-apache-flink-examples/tree/main/python/PythonDependencies)
+    leveraging `requirements.txt`.
 
 ## Reporting Bugs/Feature Requests
 
@@ -21,6 +85,7 @@ reported the issue. Please try to include as much information as you can. Detail
 
 
 ## Contributing via Pull Requests
+
 Contributions via pull requests are much appreciated. Before sending us a pull request, please ensure that:
 
 1. You are working against the latest source on the *main* branch.
@@ -31,20 +96,18 @@ To send us a pull request, please:
 
 1. Fork the repository.
 2. Modify the source; please focus on the specific change you are contributing. If you also reformat all the code, it will be hard for us to focus on your change.
-3. Ensure local tests pass.
-4. Commit to your fork using clear commit messages.
-5. Send us a pull request, answering any default questions in the pull request interface.
-6. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation.
-
-GitHub provides additional document on [forking a repository](https://help.github.com/articles/fork-a-repo/) and
-[creating a pull request](https://help.github.com/articles/creating-a-pull-request/).
+3. Update the README to reflect any changes in the code.
+4. Ensure any local tests pass.
+5. **Manually test** your changes running the application locally, in the IDE, AND in Managed Service for Apache Flink. The application must run in both without code changes.
+6. Commit to your fork using clear commit messages. Ensure you do not commit any private configuration. In particular, check the local configuration JSON file.
+7. Send us a pull request, describing your changes.
+8. Pay attention to any automated CI failures reported in the pull request, and stay involved in the conversation.
 
-
-## Finding contributions to work on
-Looking at the existing issues is a great way to find something to contribute on. As our projects, by default, use the default GitHub issue labels (enhancement/bug/duplicate/help wanted/invalid/question/wontfix), looking at any 'help wanted' issues is a great place to start.
+Refer to GitHub documentation about [forking a repository](https://help.github.com/articles/fork-a-repo/) and [creating a pull request](https://help.github.com/articles/creating-a-pull-request/), for further details.
 
 
 ## Code of Conduct
+
 This project has adopted the [Amazon Open Source Code of Conduct](https://aws.github.io/code-of-conduct).
 For more information see the [Code of Conduct FAQ](https://aws.github.io/code-of-conduct-faq) or contact
 opensource-codeofconduct@amazon.com with any additional questions or comments.

java/AsyncIO/pom.xml

@@ -17,7 +17,7 @@
         <maven.compiler.source>${target.java.version}</maven.compiler.source>
         <maven.compiler.target>${target.java.version}</maven.compiler.target>
         <flink.version>1.20.0</flink.version>
-        <flink.connector.version>4.3.0-1.19</flink.connector.version>
+        <flink.connector.version>5.0.0-1.20</flink.connector.version>
         <kda.runtime.version>1.2.0</kda.runtime.version>
         <log4j.version>2.23.1</log4j.version>
         <jackson.version>2.16.2</jackson.version>
@@ -67,7 +67,7 @@
         </dependency>
         <dependency>
             <groupId>org.apache.flink</groupId>
-            <artifactId>flink-connector-kinesis</artifactId>
+            <artifactId>flink-connector-aws-kinesis-streams</artifactId>
             <version>${flink.connector.version}</version>
         </dependency>
         <dependency>

java/AsyncIO/src/main/java/com/amazonaws/services/msf/ProcessingFunction.java

@@ -1,9 +1,10 @@
 package com.amazonaws.services.msf;
 
-import com.google.common.base.Preconditions;
 import org.apache.flink.configuration.Configuration;
 import org.apache.flink.streaming.api.functions.async.ResultFuture;
 import org.apache.flink.streaming.api.functions.async.RichAsyncFunction;
+import org.apache.flink.util.Preconditions;
+
 import org.asynchttpclient.AsyncHttpClient;
 import org.asynchttpclient.DefaultAsyncHttpClientConfig;
 import org.asynchttpclient.Dsl;

java/AsyncIO/src/main/java/com/amazonaws/services/msf/RetriesFlinkJob.java

@@ -1,7 +1,6 @@
 package com.amazonaws.services.msf;
 
 import com.amazonaws.services.kinesisanalytics.runtime.KinesisAnalyticsRuntime;
-import com.google.common.base.Preconditions;
 import org.apache.flink.api.common.eventtime.WatermarkStrategy;
 import org.apache.flink.api.common.serialization.SimpleStringSchema;
 import org.apache.flink.api.common.typeinfo.TypeInformation;
@@ -17,6 +16,8 @@
 import org.apache.flink.streaming.api.functions.async.AsyncRetryStrategy;
 import org.apache.flink.streaming.util.retryable.AsyncRetryStrategies;
 import org.apache.flink.streaming.util.retryable.RetryPredicates;
+import org.apache.flink.util.Preconditions;
+
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
 

java/AvroGlueSchemaRegistryKafka/pom.xml

@@ -53,7 +53,7 @@
             <scope>provided</scope>
         </dependency>
 
-        <!-- Amazon Managed Service for Apache Flink (formerly Kinesis Analytics) runtime-->
+        <!-- Library to retrieve runtime application properties in Managed Service for Apache Flink  -->
         <dependency>
             <groupId>com.amazonaws</groupId>
             <artifactId>aws-kinesisanalytics-runtime</artifactId>
@@ -66,6 +66,7 @@
             <groupId>org.apache.flink</groupId>
             <artifactId>flink-connector-base</artifactId>
             <version>${flink.version}</version>
+            <scope>provided</scope>
         </dependency>
         <dependency>
             <groupId>org.apache.flink</groupId>

java/AvroGlueSchemaRegistryKinesis/pom.xml

@@ -52,13 +52,14 @@
       <scope>provided</scope>
     </dependency>
 
-    <!-- Kinesis Data Analytics -->
+    <!-- Library to retrieve runtime application properties in Managed Service for Apache Flink  -->
     <dependency>
       <groupId>com.amazonaws</groupId>
       <artifactId>aws-kinesisanalytics-runtime</artifactId>
       <version>${kda.runtime.version}</version>
       <scope>provided</scope>
     </dependency>
+
     <dependency>
       <groupId>com.amazonaws</groupId>
       <artifactId>aws-kinesisanalytics-flink</artifactId>
@@ -70,6 +71,7 @@
       <groupId>org.apache.flink</groupId>
       <artifactId>flink-connector-base</artifactId>
       <version>${flink.version}</version>
+      <scope>provided</scope>
     </dependency>
     <dependency>
       <groupId>org.apache.flink</groupId>

java/CustomMetrics/pom.xml

@@ -17,7 +17,7 @@
         <maven.compiler.source>${target.java.version}</maven.compiler.source>
         <maven.compiler.target>${target.java.version}</maven.compiler.target>
         <flink.version>1.20.0</flink.version>
-        <flink.connector.version>4.3.0-1.19</flink.connector.version>
+        <flink.connector.version>5.0.0-1.20</flink.connector.version>
         <kda.runtime.version>1.2.0</kda.runtime.version>
         <log4j.version>2.23.1</log4j.version>
         <jackson.version>2.16.2</jackson.version>
@@ -37,11 +37,12 @@
     </dependencyManagement>
 
     <dependencies>
-        <!-- Amazon Managed Service for Apache Flink (formerly Amazon Kinesis Data Analytics) runtime-->
+        <!-- Library to retrieve runtime application properties in Managed Service for Apache Flink  -->
         <dependency>
             <groupId>com.amazonaws</groupId>
             <artifactId>aws-kinesisanalytics-runtime</artifactId>
             <version>${kda.runtime.version}</version>
+            <scope>provided</scope>
         </dependency>
 
         <!-- Apache Flink dependencies -->
@@ -70,10 +71,11 @@
             <groupId>org.apache.flink</groupId>
             <artifactId>flink-connector-base</artifactId>
             <version>${flink.version}</version>
+            <scope>provided</scope>
         </dependency>
         <dependency>
             <groupId>org.apache.flink</groupId>
-            <artifactId>flink-connector-kinesis</artifactId>
+            <artifactId>flink-connector-aws-kinesis-streams</artifactId>
             <version>${flink.connector.version}</version>
         </dependency>
         <dependency>

java/DynamoDBStreamSource/README.md

@@ -0,0 +1,45 @@
+# DynamoDB Streams Source example
+
+* Flink version: 1.20
+* Flink API: DataStream API
+* Language: Java (11)
+* Flink connectors: DynamoDb Streams Source
+
+
+This example demonstrate how to use Flink DynamoDB Streams source.
+
+This example uses the `DynamoDbStreamsSource` provided in Apache Flink's connector ecosystem.
+
+### Pre-requisite set up
+
+To run this example, the following resources needs to be created:
+1. A DynamoDB table - the example uses a table schema documented using `@DynamoDbBean`. See `DdbTableItem`.
+2. Set up DynamoDB Stream against the created table. See [DynamoDB Streams documentation](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html).
+3. Add items to the DynamoDB table using the schema created via console.
+
+
+### Runtime configuration
+
+The application reads the runtime configuration from the Runtime Properties, when running on Amazon Managed Service for Apache Flink,
+or, when running locally, from the [`resources/flink-application-properties-dev.json`](resources/flink-application-properties-dev.json) file located in the resources folder.
+
+All parameters are case-sensitive.
+
+| Group ID        | Key           | Description               | 
+|-----------------|---------------|---------------------------|
+| `InputStream0`  | `stream.arn` | ARN of the input stream.  |
+
+Every parameter in the `InputStream0` group is passed to the DynamoDB Streams consumer, for example `flink.stream.initpos`.
+
+See Flink DynamoDB connector docs](https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/dynamodb/) for details about configuring the DynamoDB connector.
+
+To configure the application on Managed Service for Apache Flink, set up these parameter in the *Runtime properties*.
+
+To configure the application for running locally, edit the [json file](resources/flink-application-properties-dev.json).
+
+### Running in IntelliJ
+
+You can run this example directly in IntelliJ, without any local Flink cluster or local Flink installation.
+
+See [Running examples locally](../running-examples-locally.md) for details.
+