What I miss in Amazon Kinesis Video Streams?
4 min read, last updated on 2021-01-25
Context
After working for more than 1 year with Amazon Kinesis Video Streams with more than a sizeable workload, I have to admit that this service is one of the most underestimated ones. I know that it has a very niche and specific use case, but still, many people do not realize how core service it is for many other services available on AWS and for Amazon itself (if you want to hear/read more about it, please let me know in the comments).
What is missing?
Let’s evaluate pain points one by one, then.
Documentation
I would expect much better documentation around how so many topics, but the most painful for me was the behavior on how GetMedia and PutMedia APIs works. Long story short: for specific frequencies of pushing data, GetData API switches into long-polling from regular request-response HTTP cycle. I have established empirically that if I am pushing lower frequencies than one frame via PutMedia every ~25 seconds, it uses request-response. For more frequent pushes, the GetMedia API call becomes a long-living connection for up to 45 minutes (that information is buried somewhere inside the Github repository for either producer or consumer, which I thankfully found and that pointed me in that right direction).
You may ask: so why don’t you fix it by yourself? Documentation is available on the Github as a repository here, and you can make a pull request.
Look at the dates. Erhm, so what’s the point of fixing it if it will be shoved into a freezer? 🤦
Have you heard about Amazon KIT?
Probably not, and that’s a pity. Amazon KIT stands for Amazon Kinesis Video Streams Inference Template and its purpose and architecture is well described in the official blog post:
Analyze live video at scale in real-time using Amazon Kinesis Video Streams and Amazon SageMaker
TL;DR version is that with use AWS Fargate, Amazon Kinesis Data Streams, Amazon DynamoDB, and provided implementation you can bridge Amazon Kinesis Video Streams with Amazon SageMaker endpoints. Isn’t that cool?
It is! 😍 This project contains many exciting insights into how consumers for Amazon KVS should be implemented if you have to develop it on your own. So let’s have a look into the source code to learn more!
Erhm. There is no source code - this project is not open-sourced. I do not understand why because it could help with service adoption because, in the case of writing your own consumer, you have one example: aws/amazon-kinesis-video-streams-parser-library.
To be brutally honest - this project is overly complicated and relatively specific, plus it does not show how to operate in an environment where Kinesis Video Streams are added and removed (dynamic scaling).
Psst! 😶 I have a tip for you: Amazon KIT architecture is exposed as AWS CloudFormation template, where authors refer to the Amazon ECR registry with a Docker container. On that container, you can find a JAR. I think at this point, you know that it is easy to decompile and learn from it. It’s better than nothing, but don’t quote me on that as it’s not entirely legal.
Where is AWS PrivateLink?
Another missing point is the lack of Amazon VPC Interface Endpoint, aka AWS PrivateLink for that service. I can understand why: pushing in and out video requires really high bandwidth, and it may be cumbersome to manage. On the other hand, lack of that disqualifies this service in all environments that needs to operate with strict security requirements.
Consumption is too hard at the moment!
Last but not least: at the moment, service adoption is significantly slowed down because of all the complexity required if you have to write your own consumer. It’s a non-trivial project to write, and you are on your own (with the small help from Amazon KIT).
Current fully-managed integrations with AWS ecosystem are very limited - you can integrate Amazon KVS with Amazon Rekognition (streaming to the video part of that service, which is insanely expensive) and Amazon Connect (for streaming audio files from the phone call).
When writing the consumer, I dreamed about a much simpler version to consumer video streams with AWS Lambda or using Amazon Kinesis Data Streams. No luck for now.
What’s next?
Thankfully, as AWS Community Builder, we can talk directly with AWS teams responsible for certain services. And they are eagerly listening, which may drive updates or new directions. I have to admit that this is one of the most extraordinary benefits of that program, and the list is much longer. I will explain that in a different blog post. So I am looking forward to that cooperation!
