Work

Current Challenges

Being part of a growing and successful small company is extremely rewarding. The level of impact any one person can have is motivating for me, and keeps driving me to be better. Running a development team brings this up a few notches, since the stakes are so much higher. To build momentum and innovation, our team is focused on a few key objectives:

• Modernizing our infrastructure
• Building our new platform
• Creating new products/services

But what does that mean in the real world? Let's break down each of these to help answer that.

Modernizing our infrastructure

The web moves fast, even when most customers don't. But it doesn't matter how fast they move, it is important how fast you move. Since we host websites and manage email for our clients, I don't have the luxury of moving slowly when both the industry and the threats against us are in constant motion. Even though it is a team of one (me) managing this, I still run our infrastructure like an enterprise operation. The lessons I learned from running hundreds of servers for UHC and NASA I brought here, I just cut out the meetings and paperwork.

So my focus has been on bringing that mindset here, from our maintenance windows to performance to our disaster recovery options. When I joined the company, we had dedicated servers in a colo facility and had constant downtime. Like hours a week. It was all hands on deck for months, as the problems were spread across the database, application and server layers. After digging ourselves out of that hole, I moved our entire datacenter to a virtual one, and built a redundant and robust back end.

Today we can do maintenance and updates without any impact on our sites. I also built a static version of our sites that updates every night, so if something catastrophic were to happen, we could keep serving while trying to get our servers back online. This happened over time, but the goal has been to eliminate single points of failure wherever possible.

At the same time, I also was driving down our costs. By cutting out over priced vendors and utilizing virtualized infrastructure, I was able to slash more than 6 figures from our annual operating budget, while at the same time increasing our uptime by a couple nines and driving double digit performance improvements. Super proud of the work my team and I have done here.

Not only that, but in the security realm there has been a ton of work that nobody will ever see. I used to see those stats about bots being the majority of the load on your servers and just roll my eyes in disbelief. But it's completely true. When I deployed our load balancers and took a close look at all the traffic coming in I was floored. Bing alone was almost 10% of our traffic. We have several million visitors a month across hundreds of sites. Google, Bing and Yahoo accounted for close to 20% of the load, and that doesn't even compare to the other 'search engines' out there. And then there's the scripts looking for exploits in your network. Over the past 2 years I've been focusing on eliminating everything not human or a valid search engine from hitting our servers, and it has paid off considerably.

Now there's a whole new wave of projects on deck. Our next iteration is using Docker and Kubernetes to drive even more improvements, finally enabling us to do true git-flow, automated deployments. Of all the work I've done, this has been the hardest to get a true handle on. Going from dedicated to virtual datacenters was a cake walk compared to this. Translating everything into a scripted environment was not the issue (I do that all the time), it was the way those systems worked, from reporting to troubleshooting to deployment. Nothing like our previous environments, and not an easy hill to climb.

Today we're on the downhill side of that, deploying our new platform to it and looking for other ways to leverage the systems that we've built. The next challenge is to build out the system that will allow every developer to deploy new services by themselves, programmatically. I have no doubt we can do it, but in my mind the term 'devops' is more wishful thinking by companies rather than something alive in the wild. Most developers I've met have zero knowledge of servers, and little to no interest. Most infrastructure folks look at app development as a chore, focusing on automation scripting instead. But a girl can dream, right?

Building our new platform

In order to drive growth and transform our team into one that builds tools rather than websites, we embarked on a journey to build a new, modern platform. It's exciting and challenging to do this, but one thing that is lost on most people is that although this is a new technology, the building of it is not a technology problem. It's a people problem.

The reason for this is that we are a viable, profitable business that has hundreds of clients. It's a good problem to have. We are growing, so we have constant demands for better functionality and design. But at the same time we are resource constrained and can't necessarily hire more developers - both for the fact that bringing a new dev onboard takes months to be effective AND that we can't really justify the expense right now. The result is that we have full time (and more) work but not much leeway to build the new platform.

The solution for that is to engage the entire company to help solve this problem. The buildout of our new platform is the most important project for the company, and yet the common response to managing work is to put up walls and try to manage what's coming in. Reacting to the situation rather than being proactive. Instead of that, we are getting the entire company in on solving the problem. If sales is incented to make our platform a reality, then maybe they won't pitch the 'blue sky' solution to a client, they will manage the conversation a different way. Maybe our service team is better prepared to talk real deadlines with our clients and build better expectations.

What is perfectly clear to me is that this is challenge to making the new platform a reality, not building a bigger team or putting up bigger walls. When the whole company is rowing towards the future, we are a force much greater than our numbers.

Creating new products/services

Spend enough time in a client services organization, especially one like ours (web development), and you see first hand that the pace of change and requests is unrelenting. We deal with the entire digital presence of our clients, everything from their website to SMS to email to advertising. In order to keep up with client expectations, we have to continually evolve our product offerings. Although I only deal with a subset of this issue, it is front and center for me.

It also doesn't help much that a lot of the problems we're solving for have no clear solution in the marketplace. Real attribution is a pipe dream at this point, no matter what the ad says. True location based advertising is not either. But standing still in these areas and others isn't possible. We've been working with 3rd party vendors to figure out how to leverage data to better serve our customers, build better maps and report on all of this to our clients.

Reporting and Business Intelligence are hot items now, it seems as though hundreds of solutions have popped up in the last couple of years. I've seen my fair share of them, but have yet to find one that is able to pull in our back end data, combine with common API's and build dashboards we can integrate with our client systems. Oh, and not cost 5 figures before you even start. We seem to fit right between the area any one of these systems targets, they either go full service enterprise or self service companies. We are more like an agency, so that has been the challenge. So we're building our own solution with PowerBI instead, since that is the one that hits most of our key targets.

Another push for us is using 3rd party data to help both our advertising efforts and provide deeper insights for reporting. Since our clients are location based businesses, targeting the right audience is extremely important. We've leveraged outside data well, but one thing that is very difficult is to use that to enrich reporting. The key to that is matching across multiple datasets, and this is a challenge to say the least. The holy grail for me is using that data in real time to serve the most relevant content to engage the consumer, since our sites do not have a ton of repeat visitors and are very transactional in nature (checking hours, deals, etc.).

Alrighty then...

Hopefully that gives a little background into what I do. For years people would ask me what I did and I'd struggle to find an answer that would make sense in plain english. Usually it was met with glazed eyes searching for an escape route, sometimes utter confusion. That's cool though, we can talk about something else :)