Github's Down, What Now
Yesterday, I was finishing up my work on this blog, making some final layout decisions and pushing up my changes for deploy, when something out of the ordinary happened. At some point in my work, I tried to push to Github in order to deploy some changings to staging; and I couldn’t. Turns out, Github was down.
In case you didn’t hear about it, it was pretty big news that day. To be honest, at the time I was confused, maybe even a little shocked. I’ve definitely experienced my share of outages working in production infrastructure, so the idea of an outage is all too familar. But it was the first time I’d ever seen a widespread outage occur on Github. And thats when it occurred to me: all my deploy logic, everything I used to provide new features to my applications was all of a sudden unavailable to me. My productivity had ground to a halt. I couldn’t simply push a change to Github and let the Actions logic do its magic. I’d have to manually make that deploy; but even then, the deploy depended a Github repo to pull code from. I was stuck. I couldn’t make changes in staging, couldn’t get new features to production. Nothing was getting done.
And thats when it hit me, I had a backup in place and didn’t even realize it.
Weeks earlier, while I was thinking about setting up this blog, exploring ideas for content, I wanted to do something I didn’t see anybody else doing. Everyone linked to their Github, but no one seemed to be hosting their own code repositories. I liked the idea of having some kind of personalized space to host my public code. Somewhere I could reference on my website, send friends, employers or colleagues; but most importantly, a version control UI of my own choice. And thats when I found Gitea.
I’d heard of Gitea through a work acquitance. Beyond the open-source aspect, what I loved was the UI. Simplistic, but powerful. But it wasn’t trying to be more than a place for code. None of the fluff that Github would try to show someone when I pointed them to my user’s page. It was simply a list of my repositories, and the code I wanted folks to see. So one night, fueled up on DeathWish1, I setup my very own Gitea instance: git.jameslavender.com
So Github was down and I was refreshing their status page, wondering when I could get back to work, when it hit me. I could do a manual deploy and let my deploy logic reference my Gitea instance. So quickly, I made a new private repo on my Gitea instance and pushed up my project’s code. After making a couple of config changes to reference the new repo, I was ready to get back to test things out. I ran a check; everything looked good. And then…boom! My changes were live and I was back to work.
For the three hours that Github remained down, I pushed changes to my Gitea instance and deployed changes to my site. While I had to take one more step than I normally would to get changes out (since I no longer had my Actions workflow helping me), I wasn’t stalled completely.
As I worked, I kept an eye on Github’s status and began to wonder: How many people are stalled right now? How many teams just had to cancel their new feature deploy because Github was down? Or even worse, how many people were in the middle of a deploy while attempting to fix a broken change to their application? Losing your application’s version control is losing the life of your application.
What that day taught me was if any of your deploy logic is based around your version control, that service is your lifeblood. If it goes down, productivity comes to standstill. And yet, I’ve never heard anyone talk about this. I’m positive I’m not the first to think of this, but I rarely see it talked about. I mean, Github rarely goes down, right? So how often do you really need to think about it. But isn’t the point of high availability so that there’s no single point of failure in the capability of production? So to have high availability for a production application, the host of the code for that application also needs to be highly available.
While my Gitea instance saved me in that moment, it wasn’t a perfect workaround. My Actions workflow hadn’t been running, so the checks I have in place to catch mistakes before production rollouts weren’t operating. I had to find a workaround for this. And thats when a friend told me about Gitlab’s open-source project. But not only that, he informed me you could also set up your own Actions runners to perform your workflows on a dedicated machine. And now the lightbulb was burning bright.
I think for my next project, I’ll go after a self-hosted Gitlab & Gitlab runner instance.