添加链接
link管理
链接快照平台
  • 输入网页链接,自动生成快照
  • 标签化管理网页链接
相关文章推荐
谦和的菠萝  ·  C++ (Cpp) ...·  1 月前    · 
痴情的油条  ·  [QT] ...·  2 月前    · 
豪气的移动电源  ·  Multiple citation in ...·  4 月前    · 
乖乖的香菜  ·  node-editor · GitHub ...·  5 月前    · 

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement . We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account S3 pre-signed URL file upload not working when generatePresignedUrlRequest with CannedAccessControlList.Private #1086 S3 pre-signed URL file upload not working when generatePresignedUrlRequest with CannedAccessControlList.Private #1086 duongtinh opened this issue Mar 25, 2017 · 21 comments

My example:

               AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
		ClientConfiguration clientConfig = new ClientConfiguration();
		clientConfig.setProtocol(Protocol.HTTP);
		BasicAWSCredentials creds = new BasicAWSCredentials(accessKey, secretKey); 
		AmazonS3 s3client = new AmazonS3Client(credentials, clientConfig);
		Region region = Region.getRegion(Regions.EU_WEST_1);
		s3client.setRegion(region);
		s3client.setEndpoint(serverCeph);
		Bucket bucket = s3client.createBucket(bucketName);
		File file = new File(fileNameTest);
		AccessControlList acl = new AccessControlList();
		acl.grantPermission(GroupGrantee.AllUsers, Permission.FullControl);
	       s3client.putObject(new PutObjectRequest(bucketName, keyObject,   file).withAccessControlList(acl));
		s3client.setObjectAcl(bucket.getName(), keyObject,CannedAccessControlList.Private);
               // Generate URL
		System.out.println("Generating pre-signed URL.");
		java.util.Date expiration = new java.util.Date();
		long milliSeconds = expiration.getTime();
		milliSeconds += 1000 * 60 * 60; // Add 1 hour.
		expiration.setTime(milliSeconds);
		GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(bucketName, keyObject);
		generatePresignedUrlRequest.setMethod(HttpMethod.GET);
		generatePresignedUrlRequest.setExpiration(expiration);
		generatePresignedUrlRequest.setContentType("image/png");
		URL url = s3client.generatePresignedUrl(generatePresignedUrlRequest);
		System.out.println("Pre-Signed URL = " + url.toString());

If I using s3client.setObjectAcl(bucket.getName(), keyObject,CannedAccessControlList.PublicRead); then i can get URL and this work fine
But If I used s3client.setObjectAcl(bucket.getName(), keyObject,CannedAccessControlList.Private); then I can get URL and The URL not working

I always recived error below when I passed the URl into browsers.

<Error>
<Code>SignatureDoesNotMatch</Code>
<RequestId>tx000000000000000068415-0058d62341-5229e3-default</RequestId>
<HostId>5229e3-default-default</HostId>
</Error>

And this is URL I generated:

http://server:port/bucketname/keyobject?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20170325T081827Z&X-Amz-SignedHeaders=content-type%3Bhost&X-Amz-Expires=3599&X-Amz-Credential=HGPX8TW9RGI5LEFLARTX%2F20170325%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=76cb890603b2d9a64fd9c5c670584c73f457c08fc0a31f67c7b2a6c92265a427

Somebody can have any idea or the way that I used to run URL correctly?

@duongtinh I've been able to reproduce the issue locally - I'm taking a look to see if I can figure out what's causing it.
Thank you for also posting on SO: https://stackoverflow.com/questions/43014225/s3-pre-signed-url-file-upload-not-working-when-generatepresignedurlrequest-with

@duongtinh a bit odd but it seems the content-type is what's causing the problem. If I remove the setContentType call from the GeneratePresignedUrlRequest it works. This may actually make sense because on a GET sending the content-type in the request doesn't really make sense.

Let me know if that works for you.

Sorry, realized that I made a few other modifications to your sample code:

  • Switched the client to use pathStyleAccess
  • Removed the explicit setting of the endpointUrl - this should not be required if setting the region
  • Loaded credentials from the default credential chain - but again that's just how my setup is configured, explicitly passing the credentials should also work
  • Moved to builder rather than calling setters (though this shouldn't made a difference, it's just the preferred style)
  • Full working sample here:

    ClientConfiguration clientConfig = new ClientConfiguration();
    clientConfig.setProtocol(Protocol.HTTP);
    AmazonS3 s3client = AmazonS3Client.builder()
        .withClientConfiguration(clientConfig)
        .withRegion(Regions.EU_WEST_1)
        .withPathStyleAccessEnabled(true).build();
    Bucket bucket = s3client.createBucket(bucketName);
    File file = new File(fileNameTest);
    AccessControlList acl = new AccessControlList();
    acl.grantPermission(GroupGrantee.AllUsers, Permission.FullControl);
    s3client.putObject(new PutObjectRequest(bucketName, keyObject, file).withAccessControlList(acl));
    s3client.setObjectAcl(bucketName, keyObject, CannedAccessControlList.Private);
    // Generate URL
    System.out.println("Generating pre-signed URL.");
    java.util.Date expiration = new java.util.Date();
    long milliSeconds = expiration.getTime();
    milliSeconds += 1000 * 60 * 60; // Add 1 hour.
    expiration.setTime(milliSeconds);
    GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(bucketName, keyObject);
    generatePresignedUrlRequest.setMethod(HttpMethod.GET);
    generatePresignedUrlRequest.setExpiration(expiration);
    URL url = s3client.generatePresignedUrl(generatePresignedUrlRequest);
    System.out.println("Pre-Signed URL = " + url.toString());

    @kiiadi Thanks for reply.
    I have been followed your instructions, but I encountered an error :

    2629 [main] ERROR org.targets.ceph.ReadWriteFile  - Caught an AmazonClientException, which means the client encountered an internal error while  trying to communicate with S3, such as not being able to access the network.
    2630 [main] ERROR org.targets.ceph.ReadWriteFile  - Error Message: Unable to load AWS credentials from any provider in the chain
    

    I also tried to add setting of the endpointUrl and it have an error :

    Exception in thread "main" java.lang.UnsupportedOperationException: Client is immutable when created with the builder.
    	at com.amazonaws.AmazonWebServiceClient.checkMutability(AmazonWebServiceClient.java:854)
    	at com.amazonaws.AmazonWebServiceClient.setEndpoint(AmazonWebServiceClient.java:227)
    	at com.amazonaws.services.s3.AmazonS3Client.setEndpoint(AmazonS3Client.java:682)
              

    @duongtinh after using the builder you can't set the endpoint - why do you need to explicitly set the endpoint if you've set the region? If you really do need to set the endpoint you can do it via the builder with the withEndpointConfiguration method - you need to provide both the endpoint and the signing region (you do not need to call withRegion separately in this case).

    RE: the error about loading credentials, my example loads them via the default credential provider chain. You can change this behaviour again via a builder method. Try replacing the client construction code above with this:

    AmazonS3 s3client = AmazonS3Client.builder()
        .withClientConfiguration(clientConfig)
        .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(serverCeph, "eu-west-1"))
        .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secretKey)))
        .withPathStyleAccessEnabled(true).build();

    Where serverCeph, accessKey and secretKey are the same values as in your original example.

    AmazonS3 s3client = AmazonS3Client.builder()
    .withClientConfiguration(clientConfig)
    .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(serverCeph, "eu-west-1"))
    .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secretKey)))
    .withPathStyleAccessEnabled(true).build();

    URL generated :
    http://serverName/bucketName/objectName?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20170329T090454Z&X-Amz-SignedHeaders=content-type%3Bhost&X-Amz-Expires=3599&X-Amz-Credential=HGPX8TW9RGI5LEFLARTX%2F20170329%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Signature=ec3f4096376e2fd45b7bd91079f66da9eec1d5cf62c82e6f66b91d1048c81a2b

    But it also not working with an error

    <Error>
    <Code>SignatureDoesNotMatch</Code>
    <RequestId>tx00000000000000009bc18-0058db7792-5229e3-default</RequestId>
    <HostId>5229e3-default-default</HostId>
    </Error>
    

    I have research and seen that the structure generates a signed URL look something like this:
    https:/serverName/bucketName/objectName?Signature=XXXXXXXXXXXXXXXXXXXXXXXXXXX&Expires=1316027075&AWSAccessKeyId=XXXXXXXXXXXXXXXXXXX
    but my generate a signed Url structure, not like this and that could be the cause of the error.
    If not I guess it is possible to run signed URL in a different way besides pass on browse and run but I not sure.
    Please help me.

    @kiiadi You mean something like that :

    ClientConfiguration clientConfig = new ClientConfiguration();
    clientConfig.setProtocol(Protocol.HTTP);

    AmazonS3 s3client = AmazonS3Client.builder()
    .withClientConfiguration(clientConfig)
    .withRegion(Regions.EU_WEST_1)
    .withPathStyleAccessEnabled(true).build();
    Bucket bucket = s3client.createBucket(bucketName);
    File file = new File(fileNameTest);
    AccessControlList acl = new AccessControlList();
    acl.grantPermission(GroupGrantee.AllUsers, Permission.FullControl);
    s3client.putObject(new PutObjectRequest(bucketName, keyObject, file).withAccessControlList(acl));
    s3client.setObjectAcl(bucketName, keyObject, CannedAccessControlList.Private);

    // Generate URL
    System.out.println("Generating pre-signed URL.");
    java.util.Date expiration = new java.util.Date();
    long milliSeconds = expiration.getTime();
    milliSeconds += 1000 * 60 * 60; // Add 1 hour.
    expiration.setTime(milliSeconds);
    GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(bucketName, keyObject);
    generatePresignedUrlRequest.setMethod(HttpMethod.GET);
    generatePresignedUrlRequest.setExpiration(expiration);
    URL url = s3client.generatePresignedUrl(generatePresignedUrlRequest);
    System.out.println("Pre-Signed URL = " + url.toString());

    If right , I encountered an error :

    2629 [main] ERROR org.targets.ceph.ReadWriteFile - Caught an AmazonClientException, which means the client encountered an internal error while trying to communicate with S3, such as not being able to access the network.
    2630 [main] ERROR org.targets.ceph.ReadWriteFile - Error Message: Unable to load AWS credentials from any provider in the chain

    @duongtinh I think you're missing the credentials on the builder - you'll need to provide
    .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secretKey)))

    @kiiadi So sorry, I missed

    .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secretKey)))

    But it also not working.

    @kiiadi My full example:

    ClientConfiguration clientConfig = new ClientConfiguration();
    clientConfig.setProtocol(Protocol.HTTP);
    AmazonS3 s3client = AmazonS3Client.builder()
        .withClientConfiguration(clientConfig)
        .withRegion(Regions.EU_WEST_1)
        .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKey, secretKey)))
        .withPathStyleAccessEnabled(true).build();
    Bucket bucket = s3client.createBucket(bucketName);
    File file = new File(fileNameTest);
    AccessControlList acl = new AccessControlList();
    acl.grantPermission(GroupGrantee.AllUsers, Permission.FullControl);
    s3client.putObject(new PutObjectRequest(bucketName, keyObject, file).withAccessControlList(acl));
    s3client.setObjectAcl(bucketName, keyObject, CannedAccessControlList.Private);
    // Generate URL
    System.out.println("Generating pre-signed URL.");
    java.util.Date expiration = new java.util.Date();
    long milliSeconds = expiration.getTime();
    milliSeconds += 1000 * 60 * 60; // Add 1 hour.
    expiration.setTime(milliSeconds);
    GeneratePresignedUrlRequest generatePresignedUrlRequest = new GeneratePresignedUrlRequest(bucketName, keyObject);
    generatePresignedUrlRequest.setMethod(HttpMethod.GET);
    generatePresignedUrlRequest.setExpiration(expiration);
    URL url = s3client.generatePresignedUrl(generatePresignedUrlRequest);
    System.out.println("Pre-Signed URL = " + url.toString());
              

    @duongtinh that works perfectly for me, I've copy/pasted the code above into my own workspace and it runs fine. Do you have any special characters in the bucket name and/or object name?

    Using the code above and a PNG from my local, I'm able to successfully generate a working presigned HTTP url. These are the parameters I'm using:

    String bucketName = "kiiadi.bucket.1086";
    String keyObject = "aws.png";
    String fileNameTest = "/tmp/aws.png";

    @kiiadi I'm using 1.11.106 but it does not root cause because I tried changed version and it also not working.

    I have to set Endpoint because all access files that I have to through the Endpoint.

    @duongtinh when you say you have to set the endpoint is this because you're going through a proxy? The presigned URL needs to be created directly against a valid S3 endpoint. The host header (ie: the endpoint) is included when calculating the signature of the request.

    In your case the pre-signed URL should always start with http://s3-eu-west-1.amazonaws.com/ (since the bucket is in eu-west-1) is that not the case? Are you able to get the presigned URL working when you do not manually override the endpoint?

    response-requested Waiting on additional info or feedback. Will move to "closing-soon" in 5 days. and removed needs-response labels Feb 25, 2020