Support for directly specifying temporary credentials

[et304383]

I use IAM roles with MFA.

Thus, I see no way to support passing the 3 values in directly:

• access key ID
• secret access key
• session token

I want to be able to pass these 3 values in directly. I dot not want the plugin trying to do the assume role call for me.

Any way to make this work?

[repeatedly]

Does your approach need to create S3 client in each data upload?
We want AWS SDK sample code for it...

[et304383]

If you're relying on the built in credential resolvers of any of the SDKs, I should be able to set the 3 environment variables and things just work. It does not.

[repeatedly]

S3 plugin uses access key and secret key when these are configured.

fluent-plugin-s3/lib/fluent/plugin/out_s3.rb

Line 412 in abe0433

when @aws_key_id && @aws_sec_key

So set these parameters and set session token to environment variable should work or need some code for AWS SDK?

[et304383]

I'm honestly not sure how you coded it, but there must be a way to specify the session token too, otherwise temporary credentials do not work.

[repeatedly]

I didn't test but here is simple patch for it.

diff --git a/lib/fluent/plugin/out_s3.rb b/lib/fluent/plugin/out_s3.rb
index bfe2574..b72342e 100644
--- a/lib/fluent/plugin/out_s3.rb
+++ b/lib/fluent/plugin/out_s3.rb
@@ -28,6 +28,8 @@ module Fluent::Plugin
     config_param :aws_key_id, :string, default: nil, secret: true
     desc "AWS secret key."
     config_param :aws_sec_key, :string, default: nil, secret: true
+    desc "AWS session token for credentials."
+    config_param :aws_session_token, :string, default: nil, secret: true
     config_section :assume_role_credentials, multi: false do
       desc "The Amazon Resource Name (ARN) of the role to assume"
       config_param :role_arn, :string, secret: true
@@ -196,6 +198,7 @@ module Fluent::Plugin
 
     def start
       options = setup_credentials
+      options[:session_token] = @aws_session_token if @aws_session_token
       options[:region] = @s3_region if @s3_region
       options[:endpoint] = @s3_endpoint if @s3_endpoint
       options[:http_proxy] = @proxy_uri if @proxy_uri

[github-actions]

This issue has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this issue will be closed in 30 days

[norve]

I didn't test but here is simple patch for it.

diff --git a/lib/fluent/plugin/out_s3.rb b/lib/fluent/plugin/out_s3.rb
index bfe2574..b72342e 100644
--- a/lib/fluent/plugin/out_s3.rb
+++ b/lib/fluent/plugin/out_s3.rb
@@ -28,6 +28,8 @@ module Fluent::Plugin
     config_param :aws_key_id, :string, default: nil, secret: true
     desc "AWS secret key."
     config_param :aws_sec_key, :string, default: nil, secret: true
+    desc "AWS session token for credentials."
+    config_param :aws_session_token, :string, default: nil, secret: true
     config_section :assume_role_credentials, multi: false do
       desc "The Amazon Resource Name (ARN) of the role to assume"
       config_param :role_arn, :string, secret: true
@@ -196,6 +198,7 @@ module Fluent::Plugin
 
     def start
       options = setup_credentials
+      options[:session_token] = @aws_session_token if @aws_session_token
       options[:region] = @s3_region if @s3_region
       options[:endpoint] = @s3_endpoint if @s3_endpoint
       options[:http_proxy] = @proxy_uri if @proxy_uri

@repeatedly
I tried this patch, but aws_session_token isn't sufficient as the aws_access_key_id and aws_secret_access_key are also updated periodically by an external agent. The way I supply credentials to td-agent is via:

  <shared_credentials>
    path /var/lib/td-agent/.aws/credentials
    profile_name default
  </shared_credentials>

The issue is that td-agent is only pulling the values from the token file only at the time td-agent is started. It's not refreshing the tokens from the credentials file or checking to see if it's expired from the expiration key in the credentials file. This is the general structure of a credentials file with temporary/rotating credentials:

aws_access_key_id     = ASIAXXXXXXXXXXXXXXXX
aws_secret_access_key = GYrp7H5xo3hv....
aws_session_token     = AGKa3PE93TF....
expiration            = 2023-08-14T23:11:19Z

All values need to be updated before sending objects to S3, or at least there needs to be a check to see if the token is expired before refreshing from ~/.aws/credentials file.