Skip to content

Upgrade consumer to the latest standards #16

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Upgrade consumer to the latest standards #16

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
0cc81ee
Use amazonlinux:2023 docker base image with python 3.9
timvlaer Feb 4, 2025
4036afe
Upgrade amazon-kinesis-client-python to v3.0.1
timvlaer Feb 4, 2025
315f5df
Use DefaultCredentialsProvider from the Java SDK v2
timvlaer Feb 4, 2025
febccfa
Make it clear that amazon_kclpy_helper generates a command and execut…
timvlaer Feb 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions consumer/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
FROM public.ecr.aws/amazonlinux/amazonlinux:2
RUN yum install python3 git java-11-amazon-corretto-headless -y
RUN git clone --depth 1 --branch v2.1.1 https://github.com/awslabs/amazon-kinesis-client-python
FROM public.ecr.aws/amazonlinux/amazonlinux:2023
RUN yum install python3-pip git java-11-amazon-corretto-headless -y
RUN git clone --depth 1 --branch v3.0.1 https://github.com/awslabs/amazon-kinesis-client-python
RUN pip3 install setuptools
RUN cd amazon-kinesis-client-python &&\
python3 setup.py download_jars &&\
python3 setup.py install &&\
Expand Down
7 changes: 6 additions & 1 deletion consumer/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,12 @@
#!/bin/sh
# transform sample.properties into record_processor.properties
python3 /usr/bin/set_properties.py

chmod 777 /usr/bin/record_processor.py
`python3 /usr/local/lib/python3.7/site-packages/samples/amazon_kclpy_helper.py --print_command --java /usr/bin/java --properties /usr/bin/record_processor.properties --log-configuration /usr/bin/logback.xml` &
command=$(python3 /usr/local/lib/python3.9/site-packages/samples/amazon_kclpy_helper.py --print_command --java /usr/bin/java --properties /usr/bin/record_processor.properties --log-configuration /usr/bin/logback.xml)
# execute the generated command in the background (&)
eval "${command}" &

mkdir -p /app/logs
touch /app/logs/record_processor.log
exec tail -F /app/logs/record_processor.log
10 changes: 6 additions & 4 deletions consumer/sample.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,16 @@ streamName = STREAM_NAME
applicationName = APPLICATION_NAME

# Users can change the credentials provider the KCL will use to retrieve credentials.
# The DefaultAWSCredentialsProviderChain checks several other providers, which is
# Expected key name (case-sensitive):
# AwsCredentialsProvider / AwsCredentialsProviderDynamoDB / AwsCredentialsProviderCloudWatch
# The DefaultCredentialsProvider checks several other providers, which is
# described here:
# http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html
AWSCredentialsProvider = DefaultAWSCredentialsProviderChain
# https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/auth/credentials/DefaultCredentialsProvider.html
AwsCredentialsProvider = DefaultCredentialsProvider

# Appended to the user agent of the KCL. Does not impact the functionality of the
# KCL in any other way.
processingLanguage = python/3.6
processingLanguage = python/3.9

# Valid options at TRIM_HORIZON or LATEST.
# See http://docs.aws.amazon.com/kinesis/latest/APIReference/API_GetShardIterator.html#API_GetShardIterator_RequestSyntax
Expand Down