How To Setup Firehose DeliveryStream In Localstack

AWS provides us a tonnes of services to reduce development and maintenance hassle. Those who need to stream and store(maybe analyze too!) lots of data in real-time, they can avail kinesis services. Of-course if you want to build your own pipeline then no one is stopping you… but there goes a say:

If I have seen further than others, it is by standing upon the shoulders of giants

In my case it was just putting the data to kinesis data firehose, which will periodically dump those data after some threshold in an s3 bucket. Pretty basic need. But after I started getting my hands dirty I faced some difficulties and couldn’t found any complete guide for my requirement thus I thought keeping this as a reference for my future works.

version: "3"

services:
localstack:
image: localstack/localstack
container_name: localstack-firehose-s3
restart: always
ports:
- "4567-4597:4567-4597"
- "8010:8010"
environment:
- SERVICES=s3,firehose,iam
- DEBUG=1
- DATA_DIR=/tmp/localstack/data
- PORT_WEB_UI=8010
- LAMBDA_EXECUTOR=docker
- KINESIS_ERROR_PROBABILITY=0.01
- DOCKER_HOST=unix:///var/run/docker.sock
- DEFAULT_REGION=ap-southeast-1
docker-compose up 
[default]
aws_access_key_id = 123456
aws_secret_access_key = 123456
aws --endpoint-url=http://localhost:4572 s3 mb s3://s3-firehose --region ap-southeast-1
aws --endpoint-url=http://localhost:4572 s3api put-bucket-acl --bucket s3-firehose --acl public-read
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1572416334166",
"Action": "*",
"Effect": "Allow",
"Resource": "*"
}
]
}
aws --endpoint-url=http://localhost:4593 iam create-role --role-name super-role --assume-role-policy-document file://$PWD/iam_policy.json
{
"Role": {
"Path": "/",
"RoleName": "super-role",
"RoleId": "519cuuk49exij5eims36",
"Arn": "arn:aws:iam::000000000000:role/super-role",
"CreateDate": "2019-11-08T17:19:55.050Z",
"AssumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Stmt1572416334166",
"Action": "*",
"Effect": "Allow",
"Resource": "*"
}
]
},
"Description": "None"
}
}
{
"DeliveryStreamName": "s3-stream",
"DeliveryStreamType": "DirectPut",
"S3DestinationConfiguration": {
"RoleARN": "arn:aws:iam::000000000000:role/super-role",
"BucketARN": "arn:aws:s3:::s3-firehose",
"Prefix": "test-log",
"ErrorOutputPrefix": "test-error-log",
"BufferingHints": {
"SizeInMBs": 1,
"IntervalInSeconds": 60
},
"CompressionFormat": "UNCOMPRESSED",
"CloudWatchLoggingOptions": {
"Enabled": false,
"LogGroupName": "",
"LogStreamName": ""
}
},
"Tags": [
{
"Key": "tagKey",
"Value": "tagValue"
}
]
}
aws --endpoint-url=http://localhost:4573 firehose create-delivery-stream --cli-input-json file://$PWD/firehose_skeleton.json
{
"DeliveryStreamARN": "arn:aws:firehose:ap-southeast-1:000000000000:deliverystream/s3-stream"
}
pip install boto3
<Contents>
<Key>test-log/f4b96380-000e-4442-aec0-c4f271d00c1d</Key>
<LastModified>2019-11-08T18:05:47.324Z</LastModified>
<ETag>"2dfdb4d8ce3768f4fb73e7991c59bbf8"</ETag>
<Size>52</Size>
<StorageClass>STANDARD</StorageClass>
<Owner>
<ID>
75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a
</ID>
<DisplayName>webfile</DisplayName>
</Owner>
</Contents>