In the thirty-plus years I’ve worked with analytics organizations, management has faced several recurring challenges. Without an active (even if informal) practice to watch and react to these challenges, management will find their organizations becoming unable to cope with the ever-rising flood of requests that requires a capable, agile, powerful, happy and healthy analytics team. These Top Ten challenges are sorted in increasing order of strategic impact (i.e., from least to most), and in order of difficulty of detection and prevention (i.e., easiest to most difficult).
- Bearing no responsibility for data provisioning: Data is the fuel with which the analytics engine combusts; no fuel, no model, no impact. Despite this obvious connection, some immature analytics organizations commit the mistake of expecting that the data their organization will use needs to be sourced, somehow magically, by their IT counterparts, with a minima of planning and clear requirements specification.
Solution 1: Assign staff to work with IT to accelerate the delivery of useful data into the analytics processes. At a minimum, analytics management needs to assert an active role in planning and quality assurance over the role that IT plays in providing data. At the optimal, analytics management needs to be part of the provisioning process, with multiple, high latency feedback loops and constant communication with their IT peers. In general, IT supplies data throughout the business, and while they strive for the ideal deliverable, they have resource constraints, and they are not mind readers. Analytics resources working on data provisioning have the added benefit of learning a lot more about the context and quality of the data they oversee (see points 3 and 4 below).
- Bearing no responsibility for tools of the trade: If data is the fuel of the analytic engine, then the tools are the drivetrain and the (literal) dashboard instruments. And yet, like the challenge presented in point 1 above, immature analytics organizations sometimes insist that IT provides them with the tools of the analytic trade, giving up the active role in procurement and purchasing. This sometimes occurs because analytics management, through inexperience with the practical requirements of the analytics role, is not capable of making sound trade-offs between functional and process requirements and the buy vs. build decisions accompanying capital investments. Furthermore, inexperienced analytics managers believe the delivery of tools ends with their installation; more experienced managers recognize that a concomitant investment in configuration, testing, documentation and training of their staff, tied to the specific requirements of the tool in the business process, will produce superior returns on the capital investment and in the adoption of the tool in the analytics process.
Solution 2: Inexperienced analytics management should proactively reach out to specialists that can assist with these decisions.
- Mistaking data quality as IT’s job: Data quality processes are not sexy. But they are vital to the success of the analytics initiative that relies on satisfactory data. Note the use of the term “satisfactory” in the previous sentence, as opposed to “clean”. Satisfactory implies the recognition that no data is ever perfectly clean, but that the business process using the data to derive insight is also fraught with uncertainty; therefore, an investment in data quality will help at least minimize the likelihood that an important decision will be made in error due to a fundamental flaw with the business data.
Solution 3: Invest time and energy in exploratory data analytics to understand the scope, meaning and limitations of the data and to inform the analytic organization about the nature of data quality issues that are observable and manageable. Data quality is an evolutionary process in which the business needs to actively invest time and energy; it is not a technology project.
- Not understanding the process generating the data: This phrase is common to introductory statistics academic texts, but most professionals forget the phrase when they leave college and start their first job. There’s a good reason this phrase is hammered into the heads of statistics students: the data is not the business process, it’s merely a reflection of it.
Solution 4: The analyst needs to understand the business process first; go talk with the folks in the field who understand the customer first-hand, and who often are the people actually entering the data into the system for capturing the records that end up in the corporate data warehouse. This will drive the design of a successful analytics initiative when it comes time to implement it in the field, and it will help drive the support of the people in the field who will use the analytic models, even if they don’t understand the Greek-letter equations used to derive the statistical model based on the observed data.
- Building an unhealthy relationship with IT: This is typical in most firms, but shouldn’t be. Collaboration between IT and business relationship produces benefits that are potentially enormous. And yet, this seems not to happen as often as it should. There is no ally in the organization more potentially valuable to the analytics manager than their IT counterparts. Few practices in the firm require more from IT in terms of data intensity when it comes to developing a model; one practice that is more data-intensive and systems-intensive is deploying that same model into production and then into the field.
Solution 5: Spend more time, literally, in your IT business partner’s shoes. Ask to attend some of their internal meetings where topics that affect the analytics team are of direct interest. Develop familiarity with the resource allocations and constraints that your IT partner deals with on a daily basis, and find ways to meet their constraints with some of your people’s skills and time.
- De-prioritizing documentation: Data scientists hate writing documentation. It involves writing in comprehensible English rather than in java, SAS or SQL. Proper model lifecycle management requires that a model evolves over time; yet the lifecycle of the modeler may be shorter than the lifecycle of the model. This means that someone new to the organization will be supporting and extending this model, which is where the real value happens. Documentation is critical for allowing this to happen without significant rework.
Solution 6: Find ways to harness the details in your analytics process so it becomes more self-documenting, and lightens the load on the analyst. This is a critical function and one that is typically not enjoyable for the analyst. By making it as easy as possible you can minimize the time they spend doing the activity and insure that documentation is done in a complete, consistent manner. Several tools are available to analysts that will help them with this effort, while requiring them to invest some time up front to learn more about version management, diagramming tools, inline code documentation utilities and visual prototyping.
- Thinking of a model as “one and done”: As with point 6 above, models evolve. The firm will probably never stop offering a given product to its customers, but the offering for that product will change over time. So why shouldn’t the firm also expect to change the offer response model and the product pricing model? This updating of models requires the attention of the predictive analytics team members, but it isn’t a typical practice in most companies. Even in the financial services industry where it is required by regulatory fiat, it is still an emerging field. It is more common to see decision scientists building the first few versions of a model, pick one for deployment, and put it into production, yet not revisit that model after three to six months have elapsed. Leading firms recognize that the decision scientists need to maintain diligent attention on the ongoing performance of their models, and spend the effort to improve them when required, even if this isn’t the sexiest or most fun project they take on.
Solution 7: Construct a model inventory platform where all new and existing models will be loaded, to support ongoing tracking and performance measurement. Require analysts to use it. Establish a process for revisiting models on a regular basis. Auditors expect to see these model inventories that meet audit and transparency requirements; a little medicine goes a long way.
- Hiring for academic credentials over business experience and well-rounded skills: Though the current trend is steering away from this habit, it is still a dominant attribute of some firms. On occasion, this is typical of the analytic organization that is managed by the quasi-academic. More often than not, the well-rounded analyst is cynically rejected for a position in the analytics team because they didn’t earn their academic credentials in the right program at the right university; this despite the possibility that the candidate has already constructed models for the company’s direct competitors that dramatically outperform the firm’s own models, through hard work and creativity.
Solution 8: Hiring, staffing, training and related long-run human capital decisions should be made by managers that are not only well-versed in the practice of analytics but also in the practice of business management.
- Encouraging a cultural divide between analytics & the business: This is related to point 8 immediately above. The same quasi-academic analytics manager mentioned above is commonly found in many organizations. It is disappointing to see situations in analytics organizations characterized by cynical rejection of fellow managers because they do not possess the disposition of analytic insight.
Solution 9: Identify and support analytics managers that can break down any barriers between the analytics function and the business function it serves. It is our position that for some managers, a superior ability to ask the right questions, to talk to the right people in the firm, and to have an inquisitive, perhaps ever-curious mind, is performing the same duties as the academically trained statistician; they simply aren’t formalizing their findings with complex equations written in code.
- Allowing analytic experts to hold their experience to themselves: Firms should actively erect barriers to protect their intellectual assets from competitors, but they should also actively break them down when they exist internally. This is especially important when analytics team members come up with a great idea but fail to share it in full with their colleagues, thus protecting their career path more than protecting the firm’s competitive position. Regardless, this habit exists and is commonplace.
Solution 10: The superior analytics manager should have the tools to recognize when this habit exists, to discourage an anti-collaborative sentiment, and to create implicit incentives for sharing.
