Structured vs Unstructured Data

 If you fill in a form and it is captured in a database, it becomes structured.  Otherwise, it is unstructured.  Why does this matter?  Because Fibre Channel’s block level storage paradigm is most effective when used with structured data.  As structured data growth goes, so goes Fibre Channel.  Over the past ten years, Fibre Channel has become a fixture in corporations wanting to wring the best possible performance from structured databases.  Going forward, where is the growth going to be?  Certainly, structured databases are not going away, in fact they will continue to grow.  However, search and social networking firms such as Google and Facebook work with close to 100% unstructured data.  As a result, their data centers do not use Fibre Channel, except for internal applications such as ERP.  Yes, even Google and Facebook have to produce reports for Sarbaines-Oxley. 

Google and Facebook typically use only the two SATA drives in the server for storage and place at least three copies of the data in separate servers to enable redundancy without requiring RAID, a high cost item used in the majority of structured data implementations.  They use a software framework call Apache Hadoop that was inspired by Google’s GFS (Google File System) and has resulted in the open source HDFS (Hadoop Distributed File System).  HDFS was designed to work with very large files; the best file size is a multiple of 64 MBs.  Clearly, this is overkill for text files, but when working with audio, images, and video files, HDFS really shines.  HDFS is typically used with HBase, another open source project modeled after Google’s BigTable and/or a data warehousing framework called Hive.  What does this mean?  Well, with the huge growth in data resulting from Web 2.0 websites where website users provide the data as opposed to Web 1.0 where the web site owner publishes the data, we expect that the largest growth will result in unstructured data. 

The Googles and Facebooks use server based storage where the data is stored on local disk drives or network attached storage targets that are built from standard servers.  We know that the search and social networking sites use homegrown storage.  The question is whether traditional data centers will move in that direction or whether they will be moved in that direction as many shift to Cloud Computing and Storage models.