Where will innovation come from next?

The past two weeks have been a blur for me as we are coming to the tail end of a massive project for a client and looking for our next big thing. The project involved redesigning a very complex website, moving it to a new server and making the switchover look absolutely transparent to all the users.  We partnered with a very innovative web and software design company out of Costa Mesa, Apogeeinvent, who did an absolutely incredible job of innovating our way through the masses of data and content.

Innovation.  That's what I was looking for in taking on this project.  We couldn't ignore what had been accomplished, but the infrastructure that got the client there was showing signs of strain. And budgetary restraints made it difficult to maintain it.  So whole chunks of the supports had to be pulled out, redesigned and plugged in without having the whole thing collapse.  In the past 24 hours the changeover was accomplished and with some minor tweaks and turns of the screws it's looking pretty good.
And during the past 24 hours I've come to realize that this project was a metaphor for our times.  Everything we counted on in our society's infrastructure is strained or failing.  What we used to do to make a living may not be available tomorrow.  We are crying out for innovation, but at the same time we have the prevailing "wisdon" that there is no budget for innovation.
I had a couple of beers with a marketing exec in the semiconductor industry yesterday to maybe find a way to help him innovate within tight budgetary constraints.  Of course the discussion cam around to social media, which he believes can help, but he doesn't know how to sell it to the bean counters.  There always has the be a monetary ROI, you know.
Then my old buddy, Rich Wallace popped up this morning on the Next Silicon Valley blg/news site about innovation cropping up out of social media metrics.  The short of it was the explanation of what social media is for.  We keep saying it's for communication and community -- which it is -- but it is so much more: its for data mining, and that data will determine what the market needs, how that need will be met, and what people can afford to pay for it.
Most of the technology industry has been enamored with what the technology can do, leaving the customer the responsibility of figuring out what it is for.  When money was loose, that was a good plan: Let the customer figure out the innovation.  But the customers don't have time to figure that out.  they are too busy trying to meet the needs of their own customers who can't figure out the products.
The data derived from social media platforms and practices is the way companies will figure out what the customers will actually buy.  
The essence of good sales technique is to figure out the real need of the customer and sell him something that will meet that need.  But in the past 20 years, technologists have created products and forced them on customers. Take electronic design automation (EDA) for example.  I was talking to a design manager for a VERY big semiconductor company who told me that 10 years ago, 75 percent of their design work was done on commercial EDA tools.  Today he said it's around 35 percent.  Why? Because the tools the EDA industry keeps trying to sell him are variants of the same tools they've been selling him for the past 10 years, not anything that he actually needs now.  So the tools he uses most are the ones the company develops internally.
(Now I know I'm going to get a lot of screaming emails telling me I don't know what I am talking about.  I'm just telling you what a design manager from a major EDA customer -- $20M tool budget -- told me.  He doesn't run all the designs for the company, but he still has a significant budget)
How does this come about? It comes from sales and marketing teams whose data is based on personal experience from a decade ago; a belief that was was true in 1999 is true today.  And the reason they believe that is the mechanism that provided information to them about changes started disappearing 10 years ago.
It you want to succeed in the current economic paradigm, you are going to have to start figuring out what your customers need and what they can pay for.  There is a whole new generation of customer out there that you have never met.  And 85 percent of them are on social networks looking for answers.
Like Fox Mulder said, "The truth is out there." So go get it.  You can't afford not to.