<Assume_role_credentials> throw MissingCredentialsError

[purva1192]

I am running fluentd in a Kubernetes cluster. When configuring the s3_out plugin, instead of providing the access_key and the secret_key, I used the <assume_role_credentials>. My s3 configuration looks like this -

<match s3.logs>
  @type s3
  @log_level debug
  <assume_role_credentials>
    role_arn arn:aws:iam::1111111111111:role/fluentd-test-role
    role_session_name Admin
  </assume_role_credentials>
  s3_bucket "sumo-log"
  s3_region "us-west-2"
  path logs/
  time_slice_format %Y%m%d%H%M
  utc
  auto_create_bucket true
</match>

However, I get the following error

2018-10-12 18:50:25 +0000 [info]: #0 starting fluentd worker pid=30 ppid=1 worker=0
2018-10-12 18:50:25 +0000 [error]: #0 unexpected error error_class=Aws::Errors::MissingCredentialsError error="unable to sign request without credentials set"
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/plugins/signature_v4.rb:72:in `sign_request'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/plugins/signature_v4.rb:112:in `apply_signature'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/plugins/signature_v4.rb:65:in `call'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10:in `call'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/plugins/retry_errors.rb:138:in `call'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/query/handler.rb:28:in `call'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/plugins/user_agent.rb:13:in `call'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/seahorse/client/plugins/endpoint.rb:45:in `call'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/plugins/param_validator.rb:24:in `call'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/seahorse/client/plugins/response_target.rb:23:in `call'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/seahorse/client/request.rb:70:in `send_request'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-sts/client.rb:486:in `assume_role'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/assume_role_credentials.rb:49:in `refresh'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/refreshing_credentials.rb:20:in `initialize'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/aws-sdk-core-3.30.0/lib/aws-sdk-core/assume_role_credentials.rb:40:in `initialize'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-s3-1.1.4/lib/fluent/plugin/out_s3.rb:426:in `new'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-s3-1.1.4/lib/fluent/plugin/out_s3.rb:426:in `setup_credentials'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluent-plugin-s3-1.1.4/lib/fluent/plugin/out_s3.rb:198:in `start'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/root_agent.rb:165:in `block in start'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/root_agent.rb:154:in `block (2 levels) in lifecycle'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/root_agent.rb:153:in `each'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/root_agent.rb:153:in `block in lifecycle'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/root_agent.rb:140:in `each'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/root_agent.rb:140:in `lifecycle'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/root_agent.rb:164:in `start'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/engine.rb:274:in `start'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/engine.rb:219:in `run'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/supervisor.rb:774:in `run_engine'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/supervisor.rb:523:in `block in run_worker'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/supervisor.rb:699:in `main_process'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/supervisor.rb:518:in `run_worker'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
  2018-10-12 18:50:25 +0000 [error]: #0 /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  2018-10-12 18:50:25 +0000 [error]: #0 /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  2018-10-12 18:50:25 +0000 [error]: #0 /var/lib/gems/2.3.0/gems/fluentd-1.1.3/bin/fluentd:8:in `<top (required)>'
  2018-10-12 18:50:25 +0000 [error]: #0 /usr/local/bin/fluentd:22:in `load'
  2018-10-12 18:50:25 +0000 [error]: #0 /usr/local/bin/fluentd:22:in `<main>'
2018-10-12 18:50:25 +0000 [error]: #0 unexpected error error_class=Aws::Errors::MissingCredentialsError error="unable to sign request without credentials set"
  2018-10-12 18:50:25 +0000 [error]: #0 suppressed same stacktrace
2018-10-12 18:50:25 +0000 [info]: Worker 0 finished unexpectedly with status 1

I should be able to access AWS without providing the access key and secret key.

Can anybody provide more details about it?

[repeatedly]

Does anyone have an idea for this error?
On my environment, <assume_role_credentials> works corretly and I first see "unable to sign request without credentials set" error.
So I'm not sure how to resolve this error.
Missing credentials information on the node?

[jiachenzhong]

I am also the same mistake, only AK+SK can use.

[Sphyrna-029]

Was a solution ever found for this? Experiencing the same issue when trying to use instance profile credentials.

[repeatedly]

We need help from AWS expert to investigate deeply.
For example, this error happens when no ~/.aws/credentials and no aws related environment variable.
I'm not sure full spec of AWS client but this comments show how to process credentials: https://github.com/aws/aws-sdk-ruby/blob/97b28ccf18558fc908fd56f52741cf3329de9869/gems/aws-sdk-core/lib/aws-sdk-core/plugins/credentials_configuration.rb#L42

[Sphyrna-029]

Solved my issue today. Did not have enough permissions for the plugin to access the bucket. Minimum requirements seem to be list bucket and put.

[repeatedly]

For that case, the error should be AccessDenied. Did you get MissingCredentialsError instead of AccessDenied with lack of permission?

[bhaveshmodiatekaplus]

After giving ListBucket and putobject permission also , we are facing same issue.
Can any one please tell minimum permission required. Thanks in advance :)

[dmanchikalapudi]

Did anyone figure out the issue? I see the same error trying to push logs to cloudwatch. I am running fluentd as a sidecar in fargate pod.

[mayankcpdixit]

Workaround I used:
I replaced Assume_role_credentials with instance_profile_credentials providing metadata ip/port for fetching default AWS creds:

<instance_profile_credentials>
  ip_address 169.254.169.254
  port 80
</instance_profile_credentials>

My pods had enough permissions. I had kube2iam(ref/more ref) installed and the pod had the right annotation.

Problem I was facing:
After ssh-ing aws s3 ls s3:// was working but the fluent-plugin-s3 plugin wasn't able to publish and threw:

unexpected error error_class=Aws::Errors::MissingCredentialsError error="unable to sign request without credentials set"

This suggests that it might not be a permission issue but some SDK usage related issue.

Hope it helps someone.
In case it still doesn't work, also check out: https://github.com/ccpgames/kubernetes-fluentd-s3

[pinkavaj]

In my case I have found out I need to disable containerSecurityContext in the helm chart from which is the fluentd installed to allow EC2 instance authentication.

[gpodila-disc]

Did anyone found the solution? Im facing the same problem but while writing into AWS Elastic search.

[pawanyoda]

I am facing also this error:
Without IAM configuration in fluentd.conf
Error:

[error]: #0 unexpected error error_class=Aws::Sigv4::Errors::MissingCredentialsError error="missing credentials, provide credentials with one of the following options:\n  - :access_key_id and :secret_access_key\n  - :credentials\n  - :credentials_provider"

After added assume role configuration:
Error:

[error]: #0 unexpected error error_class=RuntimeError error="can't call S3 API. Please check your credentials or s3_region configuration. error = #<Aws::Errors::MissingCredentialsError: unable to sign request without credentials set>"

But pod have already IAM role assigned and I can able to copy data to my bucket via aws s3 cp file s3://bucket --region us-east-1

Conf:

<match access_log>
  @type copy
  <store>
    @type elasticsearch_dynamic
    host "host"
    port "80"
    scheme http
    ssl_verify false
    type_name index_name
    logstash_format true
    logstash_prefix index_name
    logstash_dateformat %Y-%m-%d
    include_tag_key true
    time_key time
    time_key_format "%FT%T.%L%z"
    reload_connections false
  </store>
  <store>
    @type s3
    @log_level debug
    s3_bucket "bucketname"
    s3_region "us-east-1"
    path logs/
    s3_object_key_format "%{path}/${tag}/%Y-%m-%d/%{time_slice}_%{index}.%{file_extension}"
    storage_class STANDARD                
    utc true
    <buffer tag,time>
      @type memory
      chunk_limit_size 8MB
      total_limit_size 512MB
      chunk_full_threshold 0.9
      compress gzip
      timekey_wait 1m
      timekey_use_utc true
      flush_mode interval
      flush_interval 60s
      flush_thread_count 2
    </buffer>
    <format>
      @type json
    </format>                
  </store>                            
</match>

[yangjiel]

I suppose even without any credential given, the plugin should not crash the fluentd.

[sparampalli]

Do we have any solutions for this issue? Use of roles is more secure than access keys.