The current economic hardships are difficult and depressing, but there may be some silver linings. Years ago, I worked for a large retail company that was going through dramatic financial turmoil. The conditions there were similar to the conditions many companies face today. Cutting costs was a matter of survival, but not at the expense of operating the business. This was the challenge I was given as the CTO: Do more with less.
That is an intriguing challenge. Some people think that “do more with less” means make people work harder to compensate for the people that are let go. Other people think that “do more with less” means “work smarter, not harder”. If you think about it, both of these perceptions are rooted in a fundamental assumption that your existing operation is basically inefficient – that you have people wasting time or you have people working on the wrong things or you have people following bad processes. Depending upon your state of organizational maturity, all this may be true in which case you can “do more with less” by asking fewer people to work both harder and smarter. But good stewards of owner equity should always be trying to eliminate operational inefficiencies at all times, both good and bad. So what do you do if your people are already working hard, smart, and on the right things with the challenge of “do more with less”?
There is only one answer: Innovate!
Creativity really is the only answer. After you get rid of the deadwood, after you reduce the investment portfolio to the critical-few, after you fix the broken processes, you must find new and creative ways to meet the “do more with less” challenge. Only this time innovation comes with major financial constraints. It is one thing to be told, “I need you to do things dramatically better and here is a hundred million dollars to figure out how”, and quite a different thing to be told, “I need you to do things dramatically better and cut expenses as you figure out how”. It sounds like the impossible task, but actually the challenge provides an absolutely critical ingredient to innovation: it forces you to discard current thinking as an option! This is the silver lining to hard times. In good times it is very hard for any individual or group to break away from normative thinking – in fact, people who do think differently in normal times are often considered foolish or nut-cases. But in hard times, those nut-cases are taken more seriously and sometimes the only salvation companies have to turn to.
So let’s turn back the page to my struggling retail company for an example. Like most retail companies in the mid-90s, we were trying to come up with a data warehouse architecture. The most significant need by far was access to the massive amount of sales data captured at the point-of-sale terminals in all the stores which recorded the details of every sales transaction in a transaction log (TLOG), one TLOG for each store per day. For a variety of business and regulatory reasons, we had archived all TLOGs on optimal disks (CDs), but had yet to populate them in a formal data warehouse. We had all the data warehouse gurus of the day tell us what the design of the warehouse needed to look like which included things like a massive disk array, an extraction, translation, and loading tool (ETL for short), a star-schema data base management system, and so on. Everything was very expensive and therefore required a substantial business case which of course took a long time to create and the company’s financial stress began before the business case was presented. But the stress (and the initial exposure to the potential benefits of a data warehouse) only increased the need and appetite for analytical access to information. The pressure was on: do more with less.
A team was assembled to look at the problem again, this time without the data warehousing gurus. The constraints were made clear and believe it or not, the team was energized, not deflated, after hearing the challenge. After a couple brainstorming sessions, the group had discovered that the company already had a half dozen software components serving as TLOG readers in various places for different purposes, all yielding different results that caused confusion when information was compared up the line, and all costing money to maintain. They also discovered that the optical disks the TLOGS were stored on were accessible in an fast and addressable way because they resided in what amounts to as a Jukebox for storage and there was an electronic directory of what disk contained what TLOG from what store for what day. Finally, they discovered that the data center had a chunk of disk space that could be set aside (but not increased) and that we had a license to a business analytics package that included a light-weight data management capability. All this lead to a new, different, cheap solution to the problem. The idea was to allow business power users access to a front-end that would let them choose a range of stores and store-days; a TLOG reader would read the logs and load them into a data mart; then the user could conduct business analytics and when they were done, the space could be freed up for the next analytical need.
The plan was simple but had a few flaws (quickly pointed out by all the naysayers invested in the normative data warehousing solution). The primary concern about this solution was how long it would take to load all the TLOGs because the TLOG readers we had at the time all took ten minutes or more of computer time to process a single TLOG. Obviously this won’t work when trying to load thousands of TLOGs at one time. We brought in an outside independent consultant with experience with the point-of-sale software we were using and started an iterative, rapid development process. Every time he found a way to improve the speed of loading a TLOG, we’d ask him to cut it in half again. He went from 5 minutes to 1 minute, to 30 seconds, to 12 seconds, and finally to a couple seconds per TLOG within a couple weeks all by applying well-known programming techniques that were not well-known to point-of-sale vendors. He named the TLOG reader The Grinder and made it capable of parallel processing so multiple light-weight servers could grind multiple TLOGs simultaneously. In the end, the process could load hundreds of store days per minute and we never had anyone complain about how long it took to load a data mart. We replaced all the other TLOG readers with the Grinder which improved operational efficiency and data consistency both in the stores and across the company. We also gave the commercial rights to the Grinder to the consultant who ultimately used it as a cornerstone to a successful software business, saving us the cost of development and support.
For the first time ever, business analysts could quickly perform analytics with sales data. The solution wasn’t the high dollar data warehouse everyone said we needed. It didn’t provide instantaneous access for hundreds of people to all sales across all stores for all time. But it did provide quick access to a lot of sales data across selected stores to a few users at a time, and that was not only better than nothing, it was good enough all along!