Blackfynn Discover allows you to directly download datasets that are 1 GB or less. For larger datasets, we use Amazon Web Services (AWS) Requester Pays S3 buckets.

To download data or sync to another S3 bucket you'll need the following:

  1. An AWS account
  2. A bucket to place your data 
  3. The AWS command line tool

There may be costs associated when transferring data. Transfers between S3 buckets within the same AWS Region are free. Prices are subject to change by Amazon so you should check the latest pricing information available from Amazon at: https://aws.amazon.com/s3/pricing/.

Sign up for an AWS Account 

If you don't already have an account, you can create an AWS Free Tier account by visiting https://aws.amazon.com/free/.  

When you set up your account, you'll be asked:

  • To choose either a company or personal account
  • Add payment information
  • Verify your identity
  • Select a support plan 

Once you've completed the setup steps you'll receive an email to confirm your email and ensure your account is setup correctly. Once you receive your email, you can sign in to the AWS Console.

Create a S3 Bucket

After signing into the Console use the Find Services search bar to search for "S3". Select the S3 result to take you to the S3 service dashboard.

From here you can create a S3 bucket that will store data. Click the Create Bucket button to get started. 

After clicking Create Bucket, you'll be stepped through the process of creating a bucket. You'll first need to give your bucket a unique name and select a region. It's important to note that all Blackfynn Discover objects are in the U.S. East (N. Virginia) region. Keeping your buckets in the same region will help reduce any AWS fees.

Allow Access to Your Bucket 

Once you've created your bucket you'll need to allow access via AWS IAM. You can access the IAM dashboard by searching for IAM via the Services menu. From here you can create a new IAM user and assign it to your bucket or you can grant access to your root user by clicking the Manage Security Credentials button.

We suggest setting up an IAM user and not using your root account as it may make your AWS account less secure.

Transfer Data to Another Bucket 

To transfer data between AWS buckets you'll need to install and use the AWS Command Line Interface (CLI). You can learn how to install and configure the CLI for your device by following:

Once you have installed the CLI, you can transfer data from Blackfynn Discover to your bucket by typing the following in your local terminal: 

aws s3 sync s3://[discover-dataset-bucket] s3://[your-bucket] --request-payer requester

You can get the [discover-dataset-bucket] name by clicking the Get Dataset button on the dataset's page in Discover. [your-bucket] is the name of your bucket and can be obtained using the AWS Console.

It is important that you include the --request-payer requester  parameter in your call or the dataset transfer will fail.

To learn more about moving S3 objects between buckets you can visit
https://aws.amazon.com/premiumsupport/knowledge-center/move-objects-s3-bucket/.

Downloading Data to Your Local Device

You can download data to your local computer after you've installed the AWS CLI by using the AWS cp command line tool.

aws s3 cp s3://[discover-dataset-bucket] [local-path] --request-payer requester --recursive

In addition to ensuring you have included the --request-payer requester parameter, you must include the --recursive  parameter or your download will fail.

Did this answer your question?