Embedded Files
AWS S3
AWS S3
We describe the process to set up AWS S3 to connect your data to the Tenyks platform.
🚀 AWS S3 connection is available for the Pro plan.
Before you start
Before you start
📣 Please make sure you read the following points before continuing.
We need read-only access to the dataset images. We need to store metadata (e.g., thumbnails, compressed files); we can store it for you. If this is convenient for you, follow the read-only approach (Section 3.1). This is the recommended approach! 💫
(Optional) If you prefer to store metadata on your own cloud, you can:
Create a separate metadata bucket with read/write access (Section 3.2.1), OR
Create a single bucket with separate image and metadata folders on the same bucket, and provide us read/write access to that entire bucket (Section 3.2.2).
1. Expected folder structure
1. Expected folder structure
Here's the expected folder structure based on the option you choose:
read-only bucket:
read-only bucket:
Container name: {your_tenyks_data_bucket}
Within this container, organize your data as follows:
Images Directory:
{your_tenyks_data_bucket}/{your_dataset_name}/{images_directory_name}/image_n.png
Predictions File (in COCO Format):
{your_tenyks_data_bucket}/{your_dataset_name}/predictions.json
Annotations File (in COCO Format):
{your_tenyks_data_bucket}/{your_dataset_name}/annotations.json
Metadata Directory: We will set this up for you!
read-write bucket:
read-write bucket:
Container name: {your_tenyks_data_bucket}
Within this container, organize your data as follows:
Images Directory:
{your_tenyks_data_bucket}/{your_dataset_name}/{images_directory_name}/image_n.png
Predictions File (in COCO Format):
{your_tenyks_data_bucket}/{your_dataset_name}/predictions.json
Annotations File (in COCO Format):
{your_tenyks_data_bucket}/{your_dataset_name}/annotations.json
Metadata Directory:
{your_tenyks_data_bucket}/{your_dataset_name}/{metadata_directory_name} (see Sections 3.2.1 and 3.2.2)
2. Setting up S3 buckets
2. Setting up S3 buckets
We assume you have set up the necessary bucket(s). Please follow the structure described above. In case you need to brush up your AWS skills 👷, here is a fantastic article explaining how to create your first S3 bucket.
3. Create Policies in IAM
3. Create Policies in IAM
3.1 read-only bucket
3.1 read-only bucket
Inside the AWS Management Console navigate to IAM.
Under the 'Access management' menu, click on 'Policies'.
A list of all the available policies will be displayed, see Figure 1. Click on 'Create policy'.
In the 'Policy Editor,' select JSON.
Copy and paste the following permission: (i) replace {your_tenyks_data_bucket} with the name of your images bucket, and (ii) replace {images_directory_name} with the name of your folder for images.
JSON
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "tenyksReadOnlyAccess",
"Action": [
"s3:GetObject",
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::{your_tenyks_data_bucket}/{images_directory_name}/*",
"arn:aws:s3:::{your_tenyks_data_bucket}"
]
}
]
}
Define a policy name, and a description (optional).
After creating the policy, you should see a similar screen as shown in Figure 2.
3.2 read/write bucket
3.2 read/write bucket
📘
📘
Metadata (optional)
Metadata (optional)
Note that this read/write bucket is only required if you prefer that your metadata stays in your cloud. The recommended approach is to follow the read-onlymethod described in Section 3.1.
3.2.1 Bucket for metadata
3.2.1 Bucket for metadata
Click on "Create policy".
In the 'Policy Editor,' select JSON.
Copy and paste the following permission: (i) replace {your_tenyks_metadata_bucket} with the name of your metadata bucket, and (ii) replace {metadata_directory_name} with the name of your folder for metadata.
JSON
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "tenyksWriteAccess",
"Action": [
"s3:GetObject",
"s3:DeleteObject",
"s3:PutObject",
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::{your_tenyks_metadata_bucket}/{metadata_directory_name}/*",
"arn:aws:s3:::{your_tenyks_metadata_bucket}"
]
}
]
}
Define a policy name, and a description (optional).
Create the policy
✅ You should have two different policies, one for the images bucket and another for the metadata bucket, as shown in Figure 3.
3.2.2 Bucket for images AND metadata
3.2.2 Bucket for images AND metadata
📘
📘
Be aware: one single bucket
Be aware: one single bucket
This step assumes that
(i) your metadata will stay in your cloud
(ii) you created one single bucket with separate images and metadata directories
Click on "Create policy".
In the 'Policy Editor,' select JSON.
Copy and paste the following permission: replace {YOUR_BUCKET} with the name of your bucket.
JSON
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "tenyksFullBucketAccess",
"Action": [
"s3:GetObject",
"s3:DeleteObject",
"s3:PutObject",
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::{your_tenyks_data_bucket}/*",
"arn:aws:s3:::{your_tenyks_data_bucket}"
]
}
]
}
Define a policy name, and a description (optional).
Create the policy
✅ You should have one policy, as shown in Figure 4.
3.3 Metadata Bucket CORS Settings
3.3 Metadata Bucket CORS Settings
Go to your metadata bucket.
Navigate to "Permissions".
Scroll down to "Cross-origin resource sharing (CORS)".
Click "Edit" and "Paste" the following COR settings.
JSON
[
{
"AllowedHeaders": [],
"AllowedMethods": [
"GET"
],
"AllowedOrigins": [
"*"
],
"ExposeHeaders": []
}
]
✅ You should have the CORS settings, as shown in Figure 5.
4. Create a new User
4. Create a new User
4.1 Create user
4.1 Create user
Inside the AWS Management Console navigate to IAM.
Under the 'Access management' menu, click on 'Users'
A list of all the available users will be shown. Click on 'Create user', and provide a user name.
In 'Set permissions', follow the next steps
select 'Attach policies directly'
use the search box to find the 1 or 2 policies (depending on the number of buckets created) defined in section 2
select the 1 or 2 policies, and click on "Next". See Figure 6 for more details
Click on 'Create user'. A screen similar to Figure 7 should indicate your user was successfully created.
ℹ️ For a refresher on IAM roles, the following AWS docs are a great start on all you need to know about IAM roles.
4.2 Create access key for the new user
4.2 Create access key for the new user
Inside the AWS Management Console navigate to IAM.
Under the 'Access management' menu, click on 'Users'
Select the recent user created on the previous step.
Under 'Summary, click on 'Create access key'. See Figure 8 fore more details.
In 'Use case', select CLI as shown in Figure 9.
Set description tags (optional).
Click on 'Create access key'.
A screen containing the user's access key and secret access key will be displayed. Download the csv file. See Figure 10.
5. Summary
5. Summary
We successfully created the following:
One bucket for images and another for metadata(optional), OR one single bucket with independent images and metadata folders (optional).
One or two permission policies (depending on the number of buckets created).
One user configured with the two policies.
access key and secret access key for the user (to programatically access the buckets).
To conclude this procedure, kindly provide us with the following information: username, access key and secret access key.
Page updated
Google Sites
Report abuse