You may have heard the term "Big Data" before, but did you ever wonder what it means exactly, or more importantly, how it affects your life? That's the question Tech Talker answers in Part 1 of his series on Big Data.
You may have heard the term "Big Data" before, but did you ever wonder what it means exactly, or more importantly, how it affects your life? Today we'll discuss just that.
What Is Big Data?
Let's talk about what Big Data actually is. Big Data is a pretty general term used to describe huge datasets that require a massive amount of storage (measured in terabytes, petabytes, and exabytes), and huge amounts of processing power to make sense of all the data.
Many of you are probably familiar with gigabytes because this is what most of our electronics are measured in. Well, a terabyte is a thousand gigabytes. There are devices such as hard drives that can be measured in terabytes. The next order of magnitude is petabytes, which is 1 million gigabytes, and then followed by the next order of magnitude which is exabytes - that's 1 billion gigabytes.
Let's put this into perspective. One petabyte of high definition 1080p video would last for 34 years, non-stop, always playing. Two petabytes of data comprises information from all of the United States' academic research libraries.
If we take the next jump in magnitude into the exabyte range, things get even more crazy! It's pretty difficult to put any real perspective to 1 exabyte, but one statistic I came across noted that 1 exabyte would be the record of every word ever spoken by every human being that ever existed. To me, that is insane. To top it off, the SKA telescope initiative is predicted to produce that much data each day within the next few years.
You'll have to excuse me, I'm an engineer and I get carried away with math and large numbers.
Where Does Big Data Come From?
Your next question might be, where the heck is all of this data coming from? Well there are some research fields that generate vast amounts of data, such as radio telescopes, genomic studies, and advanced physics experiments such as atom smashing in a particle accelerator. This type of scientific research create vast amounts of data, which is then analyzed using a huge network of computers. While this form of Big Data may indirectly impact our lives through new discovery, there is another form of Big Data that gathers its information from elsewhere.
That other form of Big Data is created by us! You're probably thinking "Come on, Tech Talker, I can't possibly be creating that much data!" Well you're probably right. You as an individual are not creating such vast sums of data. However, your online habits are being tracked, cataloged, and analyzed every day.
Think of everything you buy online, your credit score, your search history, your route to work, the groceries you buy, and even your doctor visits. These are just a few of the ways you're probably contributing to Big Data without ever knowing it! Everything I mentioned is stored away in some database waiting to be analyzed. The power of this data does not lie in any single person, but in the trends that can be teased out of very large datasets.
If you want to play around with some of these huge datasets, Google has a pretty awesome feature called Google Trends, where you can put in search terms and it will show you graphs of where and when those searches took place. You can see interesting habits that occur every year, and even strange searches that pop up in certain places in the world. Check it out at Google.com/trends As an example, I recommend looking up "Shopping" and "War" to see some surprising trends.
Now that we know where Big Data comes from, the next question is: Where is it stored?