DynamoDB
This project publishes data to DynamoDB.
How to run
Create a Quix account or log-in and visit the Samples to use this project.
Clicking Edit code
on the Sample, forks the project to your own Git repo so you can customize it before deploying.
Requirements / Prerequisites
AWS access key id and secret for a user having permission to perform dynamodb:BatchWriteItem
on supplied table_name
.
Optionally, dynamodb:CreateTable
on resource arn:aws:dynamodb:region:account-id:table/quix-*
for connector to auto
create DynamoDB table for you.
Example policy for pre-created table
{
"Effect": "Allow",
"Action": [
"dynamodb:BatchWriteItem"
],
"Resource": [
"arn:aws:dynamodb:region:account-id:table/my_table"
]
}
Example policy for creating table automatically
{
"Effect": "Allow",
"Action": [
"dynamodb:CreateTable",
"dynamodb:BatchWriteItem"
],
"Resource": [
"arn:aws:dynamodb:region:account-id:table/quix-*"
]
}
Environment Variables
The code sample uses the following environment variables:
- input: Name of the input topic to listen to.
Required
- table_name: Name of the Amazon DynamoDB table, same as input if not provided.
- aws_access_key_id: AWS access key id.
Required
- aws_secret_access_key: AWS secret access key
Required
- max_retry: Max reties for failed batch of records
- param_partition_key: Parameter name to be used as partition key.
- param_sort_key: Parameter name to be used as sort key.
Using param_partition_key and param_sort_key
Environment variables param_partition_key
and param_sort_key
can take any of below values:
- Direct string type parameter name from the message OR
- Derived key attribute name and strftime
pattern joined
by pipe
|
. (e.g.hour_of_day|%Y-%m-%d %H
)
For example message,
{
"timestamp": 1667844200703123, // 2022-11-07 19:03:20.703123
"key1": "value1",
"key2": 10,
"key3": "value3"
}
Example 1
# Note: Only string type parameter is allowed as parition key and sort key at the time.
param_partition_key = key1
param_sort_key = key3
DynamoDB Record
key1 (PK) | key3 (SK) | timestamp (N) | key2 (N)
value1 | value3 | 1667844200703123 | 10
Example 2
param_partition_key = year_month|%Y-%m
param_sort_key = rest_of_ts|%d-%H:%M:%S.%f
DynamoDB Record
year_month (PK) | rest_of_ts (SK) | timestamp (N) | key1 (S) | key2 (N) | key3 (S)
2022-11 | 07-19:03:20.703123 | 1667844200703123 | value1 | 10 | value3
Example 3
param_partition_key = my_date_time|%Y-%m-%dT%H:%M:%S.%f
param_sort_key = (empty)
DynamoDB Record
my_date_time (PK) | timestamp (N) | key1 (S) | key2 (N) | key3 (S)
2022-11-07T19:03:20.703123 | 1667844200703123 | value1 | 10 | value3
Contribute
Submit forked projects to the Quix GitHub repo. Any new project that we accept will be attributed to you and you'll receive $200 in Quix credit.
Open source
This project is open source under the Apache 2.0 license and available in our GitHub repo.
Please star us and mention us on social to show your appreciation.