Chatting with one of my friends who is working on some very cool cloud products in a very entrepreneurial team, he regaled me with his field support challenges,
Sorry, I’ve had a horrible few weeks – I’ve been bouncing back and forth between 2 cities fixing up (name redacted) cloud, just got back again last night.
The long and the short of it is that the original deployment was done by 2 guys who I wouldn’t trust to administer my iPad let alone a cloud, once I worked through that I went to bring the block storage on line and realized they’d screwed that up as well. So I rebuilt the cloud, the ran their deployment automation against it, broke it, I fixed it, they broke it again, I fixed it again and told them to stop breaking it.
So hilariously, this only came about because the onsite resources didn’t have their shit together, in the words of the customer ‘they keep calling you to fix stuff.”
It’s that personal scaling issue again. You want a life then the customer needs to be able to trust more than just you. I know I sounded pretty much like a process wonk to many youngin’s out there, but fact is that I really do know how to do just that scaling, and my former employer’s infrastructure group was doing it so wrong ‘cause the management didn’t even understand the anchor they were dragging around.
But it does require a little bit more investment than folks who are focused on expediency are usually willing to give. As a result folks like you get to rack up frequent flier miles to the same site over and over, projects fall behind, blah, blah, blah. Of course if the goal is to just get up on the map as fast as possible then well, I’m not so interested since that would be wasting my main skills.
Yep, know that one well. If it happens a lot it bogs down the group and as a bonus, pisses off the customers anyway. If it keeps going the customers begin to see the whole thing as half-assed and they begin to bail out.
In the early 90’s the company where I learned abut software engineering had all kinds of informal backdoors to developers from various professional relationships with both customers and field support folks. It got so bad that it was hobbling our development velocity. Part of the reason was intellectual laziness out in the field but a bigger reason was an accumulation of difficult, deep problems with our products or processes. All this scooting around was really pissing our customers off. We had a lot of product drop threats from major international oil company customers. We were fixing little stuff but were clueless about the real root causes.
I was a lead Program & Product Manager at the time. I visited customers a lot so I knew about this more than I ever wanted. I had a huge feature backlog already in place so for a change I pitched the creation of a hot team of our best support folks as a 2nd tier support group, reporting up through the development management rather than our Customer Support group, along with a weekly worldwide issues meeting with all the regional VPs & ops Directors in the home office. I was on a roll at that company so they gave it to me…my 2nd management spot. I ran the meeting with a strict agenda & a 1 hour time cap, and we met at 6am central time. I had 3 of our best support geoscientists to cover an escalation process plus mining the customer support DB for repeated issues which we worked with the devs to fix once and for all. Over a 2 year period we eliminated all of the known chronic & design caused issues, directly addressed specific problem customer issues, and raised our customer satisfaction up from a low of 48% to around 80%. It kept our market share up at 70% so it worked like a charm.
But it cost a lot. We had to staff this with 3 of our best Program Managers & ultimately 2 more staff to replace our old Customer Support & Bug tracking system. While we were at it we set up the company’s first web site . This FTE Staffing, plus the capex for those tools, and of course 20 hours a week from about 20 very high paid people around the world.
Finally after 2 years of shoveling through the technical debt, we dissolved the team & rolled the processes into the Support Group since we’d made all the good stuff routine & I got bored with running it.
The important thing though is that if we had the escalation structure & processes in place before issues started to stack up and fester we would have had to spend a tiny fraction of the costs we had by letting them pile up. We learned that because once we got it all stabilized we could plainly see the drop in costs.
So if your business plan is planning to require big scale-ups of your customer base, then it makes sense to start this kind of framework early and make it scalable too so it can start small and grow only as needed.