The Bigger Picture: Data Scale, Speed, Diversity and Complexity

It’s a privilege to work with the unsung heroes of our generation: the legions of IT professionals who tirelessly build, maintain, and innovate around the infrastructure that makes our digital existence possible. Believe it or not…interacting with IT professionals, hundreds a year, at every level, in all functions, across multiple industries, and all seven continents– is a ton of fun!

Here’s what they are saying about their priorities, nearly unanimously. Two words: business and security.

Keep Business Booming
More than ever, taking care of the business stakeholder is job one. Every system that is revenue generating or customer-facing is a top priority, like the .com and mobile user experience. Keeping performance high and costs down across the data center and network are also a top priority. IT pros in both development and operations have been feverishly at work blowing the doors off their internal business stakeholder’s expectations. They’re executing on the CEO’s digital user experience strategy in mobile and social, and helping CROs greasing the wheels of e-commerce with big data. IT architects are driving higher performance and lowering costs with cloud, hybrid, and converged infrastructure. IT Development has gone agile, bringing “new” to market at a breakneck pace using the latest rapid application development tools. IT Operations is managing unprecedented scale and complexity with essentially flat budgets. The global business cycle is fully recovered in nearly every industry. Consumers are back in droves, especially on digital channels. Balance sheets are cash heavy. Profits and market caps are hitting all time highs. Business is booming and competition is fierce, due in large part to the relentless innovation that IT has driven over the last decade thru the global recession and recovery.

Keep Business Secure
Breached is the new normal. Denial of service attacks used to be the top bullet, lately breach is #1. But not for the reasons you might think. Hard financial losses are far less of an issue than lost consumer confidence, negative brand image, and career-arrestors associated with a breach. The initial scary estimates of the mega-breaches at Sony, Target, and Home Depot were in the billions of dollars. But actual costs were less than 1% of revenues, so incremental IT spend will be minimal. However, lost confidence, and the embarrassing black eye of the perception of incompetence on the part of senior management and other IT professionals, makes any comparable endeavor invested to prevent breaches seem well worth the effort. In Target’s case, not only the CIO, but the CEO was replaced, and the board was severely challenged. Ironically, this lack of a financial model to invest exponentially more in IT security creates severe pressure on both business and IT leaders to harden against breaches by using smarts and available levels of current spend. Knowing what I know, I am an optimist. Monitoring the IT environment for breach activity above the noise is effective and efficient, and cost no more when incorporated into an existing infrastructure monitoring architecture.

The Bigger Picture
IT professionals talk about a few titanic shifts that have happened in the last decade that I will summarize in four concepts: scale, acceleration, diversity, complexity.

Scale. Disruptive maturation of social, mobile, cloud, and big data in the last decade has left an indelible new mark on the IT industry: scale. It’s not big data anymore, it’s just data. The scale of data is boringly hyper-exponential. The new terabytes is petabytes. There is so much data in the world we’ve stopped trying to estimate the totality of it, and go for daily creation of data instead, estimated at 2.5 quintillion bytes a day. That’s 18 zeros, every day. And it’s getting faster.

Speed. Moore’s Law and a digitally-native, Generation Z, makes increasingly faster proliferation of data scale a certainty for the foreseeable future. It may as well be 100x 18 zeros/day, and the infrastructure needed to support more data will spring up overnight from the same virtuous business innovation cycle that created the demand in the first place. We have changed too. Far from being crippled by infobesity, as many predicted, we are all too comfortable with more data. Generation Z, the first to know the world as purely digital, expects even more data in their lives, as technology has abstracted data and hidden its messiness and complexity from view. Mere data, as such, is too abstract for most to discern the impact of its immeasurable scale and exponential growth rate.

Diversity. Marry an intensely competitive virtuous business cycle with equally impressive exponential progressions in science and innovation, mix in $100B annually in venture capital — and viola, whole new industries were birthed in the last decade that have yet to make their mark on the IT landscape. These toddlers will each grow to be behemoths in their own right, which will both consume and enable our appetites for data: IoT, blockchain, wearables, autonomous transportation, cognitive systems and AI, 3D printing, smart healthcare, renewable energy, nanotechnology, domestic robotics, smart education, smart ag. Data diversity drives data complexity.

Complexity. Business leaders hear that dramatic advances in virtualization, cloud, and converged infrastructure have propelled data at scale into competitive advantage for a diverse range of businesses. While true, IT professionals see the “real” impact of data scale, speed, and diversity: inevitably more hardware. IT on a “virtual machine” or “in the cloud” eventually has to actually reside on a very real piece of hardware in a very real data center somewhere on our tiny blue planet. It takes real capital to invest in that infrastructure. It places real burdens on our environment as hardware consumes scarce precious metals, burns real energy that increase carbon footprint, and occupies the attention of real human beings who need to manage and maintain it. If data scale is growing exponentially with no end in sight, so too are the real headaches of modern IT infrastructure. So counting actual hardware, real ‘metal’ vs abstracted data, is useful. Like cars, the more of them on the planet, the more impactful the consequences, positive and negative. Last year alone, 10 million servers were sold, more than ever before. Like cars, the fastest growth rates are seen in developing economic regions like Asia. No one knows for sure, but Google, Microsoft, Amazon, and Facebook are all estimated to have a million+ physical servers each. This is the new IT, and it’s messy. Complex is the new normal. It is here to stay.

So there you have it. If you are in IT, here’s what your boss is asking you to do today: keep business growth humming, and protect increasingly complex digital assets from cyber-threats. Why? The bigger picture is: data scale, speed, diversity, and complexity.

Like the Zenoss blog? Please provide your email address below to subscribe and enjoy reading updates right from your inbox!