Posts

Amazon DynamoDB Standard-IA vs DynamoDB Standard

Image
Recently Amazon DynamoDB Standard Infrequent Access (DynamoDB Standard-IA) table class, which AWS claims   helps you to reduce your DynamoDB costs by up to 60% for tables that store data that is infrequently accessed. In this article let me give you the price dynamics of  DynamoDB Standard-IA What DynamoDB costs break down Data storage Write traffic Read traffic Other Costs On-demand backup Continuous (PITR) backup Table restore Change data capture for Kinesis Data Streams Data export to Amazon S3 DynamoDB Accelerator (DAX) DynamoDB Streams Let's just focus on data storage costs, which is a recurring cost, that grows as your storage grows. DynamoDB storage:         20 Terrabytes DynamoDB Standard      Cost  per 1 GB-month:       $0.25   Cost per 20 Terrabytes:    20000 *  $0.25  = $5000 per month DynamoDB Standard-IA Cost  per 1 GB-month:       $0.10   Cost per 20 Terrabytes:    20000 *  $0.10 = $2000 per month *In US-East The cost slop is steep for DynamoDB Standard storage. If y

AWS S3 Bucket limitations

Image
Software systems always have some limits, reasons for limitations can varies. Hard limits boils down to limitations of underlying hardware resources. Soft limits can be to avoid noisy neighbor problems, that's where rate-limiting, service quotas comes into place. Let's explore on limitations of AWS S3 bucket   Number of S3 buckets in a AWS Account By default, you can create up to 100 buckets in each of your AWS accounts. If you need additional buckets, you can increase your account bucket limit to a maximum of 1,000 buckets by submitting a service limit increase. There is no difference in performance whether you use many buckets or just a few. TL;DR; 1000 buckets per account Cannot change S3 Bucket name or Region When you create a bucket, you choose its name and the AWS Region to create it in. After you create a bucket, you can't change its name or Region. Reuse S3 bucket name You can reuse S3 bucket name, but.. when you delete the bucket and the name becomes available for

Linux limits on threads per process

Image
I came across an interesting learning on number of threads a process can hold. # of threads per process = total virtual memory / (stack size*1024*1024) Increase in virtual memory can directly leads to increase in # threads per process. Virtual memory can be at most swap memory Increase in stack size decreases # of threads per process Check Virtual Memory:  ulimit -v (default is unlimited, thus you need to increase swap memory to increase this) Total Stack Size: ulimit -s (default is 8Mb) Command to update these values: ulimit -s newvalue ulimit -v newvalue Reference https://dustycodes.wordpress.com/2012/02/09/increasing-number-of-threads-per-process/

Gradient descent algorithm in Machine learning

Image
Machine Learning and Deep Learning commonly uses Gradient descent (GD) to find a local minimum/maximum of a given function. Gradient descent minimises a cost/loss function (e.g. in a linear regression). In mathematical terms, Gradient descent is a way to minimize an objective function J(θ) parameterized by a model’s parameters θ ∈ Rd by updating the parameters in the opposite direction of the gradient of the objective function ∇θJ(θ) w.r.t. to the parameters. The learning rate η determines the size of the steps we take to reach a (local) minimum. In other words, we follow the direction of the slope of the surface created by the objective function downhill until we reach a valley. There are three variants of gradient descent, which differ in how much data we use to compute the gradient of the objective function. Choice of variant depends on accuracy and time taken to compute parameters. Batch Gradient descent A batch gradient descent, computes the gradient of the cost function w.r.t. to

Supervised learning and unsupervised learning

Image
This article explains about basic Machine Learning concepts of Supervised learning and Unsupervised learnings.  Human learning happens with external sense objects. You can now read English because you have learned English, but if some text is in a language which you don’t know you cannot understand it. Same applies to data driven algorithms which utilizes the data to judge are loosely called Machine Learning algorithms. Let’s say, you want to develop Machine Learning algorithm to detect whether car exists in a image Supervised learning In supervised learning you'll label several images of cars and non-cars and an algorithm will eventually learn and be able to predict whether or not an unseen image has a car in it. For supervised learning the feeded data must be labeled, or explicitly mention which ones are cars and which ones aren't. When you complete a Captcha and identify the images that have cars, you’re labeling images! A supervised machine learning algorithm can now use t

Applications of Artificial Neural Network

Image
Many artificial neural network techniques have been adopted in the research, academics and industries to address the challenges in computer vision, speech and pattern recognitions, face alignment, and detection. These include; Speech recognition Artificial Neural Networks have capability successfully applied in speech or communication recognition. In the past decades, ML algorithms have applied widely in areas like acoustic modeling and automatic speech recognition. Computer vision Computer vision aims at making computers to accurately understand and process visual data efficiently like videos and images. Computer vision alludes to the logical control which studies how to separate data from images in artificial frameworks. Sub domains of computer vision include object detection and object recognition, object estimation, object position, event detection, scene reconstruction, image restoration, image editing, video enhancement, and statistical learning. Hence, in computer vision, ANN mo