How to control log settings in Spark and stop INFO messages?
January 25, 2021How does Cartesian Product Join work in Spark?
January 29, 2021You probably heard this – Amazon S3 is not a file system, it is an object storage. So in S3, there is no technical concept of a folder.
If that’s the case, why do I see folders in AWS S3 console?
Folders in S3 are meant only for organization purposes. For us to organize the objects that make sense for us. The Amazon S3 implements folder object creation by creating a zero-byte object.
If you see a file in the console you will see the key of the file also has the folder reference in the key – test-folder/hdfs-0.0.1.jar.zip. test-folder is the folder name.
Solution
We are now going to create a new folder named new-folder and upload a file into that folder.
[osboxes@wk1 ~]$ aws s3 ls s3://hirw-test-bucket PRE / PRE test-folder/ 2016-11-05 12:43:00 3411 block_-3863181236475038926
Here when we copy the file we mention the destination as new-folder/test-file eventhough new-folder doesn’t exist.
[osboxes@wk1 ~]$ aws s3 cp test-file s3://hirw-test-bucket/new-folder/test-file upload: ./test-file to s3://hirw-test-bucket/new-folder/test-file
We can now see the new-folder when we do a listing on the bucket.
[osboxes@wk1 ~]$ aws s3 ls s3://hirw-test-bucket PRE / PRE new-folder/ PRE test-folder/ 2016-11-05 12:43:00 3411 block_-3863181236475038926
We also see the folder created on the AWS S3 console also. Check out the key on the file as well.
