Update on 2011 Predictions for Data

Back in early 2011 I made a number of predictions (http://the-data-man.blogspot.com/2011/01/few-predictions-for-data-in-2011.html) in relation to the world of data architecture and data management. In this blog I'm going to revisit those predictions and see how well I scored.  So starting from the top my predictions (and results for the year) are as follows:

1/ Data Security will become a mainstream issue.

My original prediction: With the growing number of data security issues over last year such as the whole Wikileaks fiasco, the Stuxnet worm, UK child benefit details and various customer details leaks (such as the two million Honda customers that had their names, login names, email addresses and vehicle identification numbers stolen). As evidence for this the US is building the nation’s first cyber security headquarters in Utah for $1.5 billion. I expect we will see other countries follow suit.

So what happened during the year: Only a few days ago (30^th December 2011) Mark Dampster from The Centre for Cyber Security warned that businesses must have data security at the heart of their considerations for 2012. warned. He cites the recent issues at US security firm Stratfor which was the victim of a Robin Hood-style attack, in which funds from the accounts of major businesses were transferred to various charities - http://www.bcs.org/content/conWebDoc/43241.

A blog from the Washington Post’s Security Fix blog says that the Identity Theft Resource Center tracked 342 reports of data breaches between January 1 and June 27. That’s a 69 percent increase over the same period last year.

According to the Privacy Rights Clearinghouse 2011 saw some of the largest and most high profile data breaches in history. In an article in InformationWeek they list the following examples - Sony, Epsilon, RSA, Sutter Physicians Services, Tricare and SAIC, and finally Nasdaq.

To find out more read the full article:  http://informationweek.com/news/security/attacks/232301079

My score 10/10

2/ Yotta will be usurped by Hella.

My original prediction: The largest named data number is 1024 which is called the Yotta. A physic student from the University of California has a facebook campaign going to have a scientifically accepted name for the number 1027 (that is 1 followed by 27 zeroes, or 1000000000000000000000000000). Yotta (1024), which was established in 2001, is currently the largest number followed closely behind by zeta (1021), exa (1018) and peta (1015).

So what happened during the year: I don’t have an update on the campaign itself except to say that The Telegraph and the New Scientist both ran articles about this early in the year. What is worth saying is that back in February 2011 I wrote a blog about scientist calculating the amount of data in the world. In an article in the journal ‘Science’ it was calculated that the amount of data stored in the world by 2007 was 295 exabytes. The study showed that in 2000 75% of stored information was in an analogue format such as video cassettes, but that by 2007, 94% of it was digital.

My Score 5/10

3/ Hardware and software for managing data will adapt to the data tsunami from the corporate world.

My original prediction: I expect that data warehouse appliance technology will grow in importance with this market making the move from niche offering to mainstream. This will mean that open source appliance offering will become more common place and SAAS options will also become more prevalent.

So what happened during the year: Well we have IBM with its Smart Analytic System, DB2 PureScale database clusters and Netezza data warehousing appliances. Oracle is now making a lot of noise about its Exadata and Sparc SuperCluster machines. Teradata is also releasing new technology into this space.

Add into this we now have the term ‘Big Data’ starting to appear in the press. Big Data is in essence analytics against huge volumes of diverse and varied data; such as velocity, volume, variety and complexity, which comes from not just rows and columns of structured business data, but also from click streams data on websites, micro blogs, Facebook posts and others. It also is available outside the internet world across industries such as retail, automotive, telecommunication, supply chain, Internet and others through sensor devices such as cell phones, GPS, black box in vehicles, shopping cards, credit or debit cards and others. All this leads to massive amounts of complexity as I am sure you can imagine.

For a bit of a summary have a read of this article http://www.itworldcanada.com/news/big-data-analytics-get-even-bigger-hotter-in-2012/144540

My score 10/10

4/ Telecom operators will invest heavily in infrastructure.

My original prediction: The telecom industry is going through a huge transformation at present as data traffic grows. It’s a well-known fact that Data traffic overtook voice traffic long ago, and data revenues are now starting to take significant share from voice revenues. For example video streaming in the US alone has risen 78% each year since 2005. It has also been estimated that mobile data volumes will double each year until 2014. The current infrastructure is not up to the task which means more money will need to be invested becomes.

So what happened during the year: Not sure I did very well with this one as there not been any news (that I’ve noticed on the subject) if anyone knows differently let me know.

My score 1/10

5/ Mobile devices will absorb BI applications.

My original prediction: It is highly likely that BI applications (initially dashboards and reports) will start to appear on mobile devices and mobile phones. In future years I expect to see true analytical applications for mobile devices as the BI vendors target this sector in growing numbers.

What happened during the year: According to a recent Gartner forecast thirty-three percent of business intelligence functionality will be consumed through hand-held devices by 2013. Think this forecast says it all.

My score 10/10

6/ BI market will continue to grow.

My original prediction: As the global recession starts to recede BI technology will continue as a growth market. The focus will change from cost cutting to how companies can grow revenue and profits.

What happened during the year: The amount of jobs available out there in this space compared to others, the comments from the previous prediction and the amount of consolidate in the space lead me to feel that it is definitely a growth area.
Another interesting set of news headlines relates to the release of anonymised patient data by the UK NHS. The British government is allowing this because it believes that it will allow the pharmaceutical industry to grow and prosper with the UK.

My score 10/10

7/ Demand for Data Experts will grow but will be hampered by available resource.

My original prediction: Over the last decade the demand for data architecture skills has increased. This demand will be held back by the shortage of resources in this area that have the necessary skills. There are many database experts but database knowledge doesn’t make a data architect (see my blog on the role of the data architect). A data architect needs to be able to see the big picture as well as the detail. Without the enterprise view a data architect fails in his/hers fundamental role of growing the value of the organisation data assets.

What happened during the year: I have a number of touch points on this to justify a high score. Firstly an article by Cliff Saran in his Computer Weekly blog (http://www.computerweekly.com/blogs/it-fud-blog/2011/12/its-scarce-differentiators-in-.html) talks about the importance of data architecture as the CIO’s new differentiator.

To quote Saran “There's no point in building a better data analytics tool if there isn't a fundamentally sound foundation in place: a proper data architecture, with governance rules, master data management and a scalable storage architecture. Without this critical base, higher order analytics simply aren't possible. Unfortunately, this is typically one of the areas where firms are worst prepared, with data silos and platforms that aren't able to communicate with each other. Data architects will become increasingly crucial as firms become more reliant on data to compete.” There are many other articles on this subject in papers such as Forbes, FT and many blogs and discussion boards.

I’ve recently launched my own course on data architecture in an attempt to address this skills shortage. The first course was run in October and produced some great feedback. For anyone who is interested the next course is scheduled for the 8^th and 9^th of March (http://www.koios-associates.com/DataArchitectureCourse.htm).

Therefore I feel confident in a awarding myself a high score for this one.

My score 10/10

8/ Data Governance will be a hot topic.
My original prediction: Data governance has already become, and will just continue in this vain, as a hot topic. With bank risk management, data security and many other data related issues becoming board level subjects this is guaranteed to increase in importance.

What happened during the year: This is definitely a hot topic, for example recently Bank of America hired John Bottega to the role of chief data officer (CDO). In this position, Bottega is responsible for the banks's data management strategy, policy and governance. http://www.banktech.com/management-strategies/232300494

My score 10/10

So an overall score of 66/80. In a few days (possible a few weeks) I’ll put up my predictions for this year.