ChatOps at GitHub - Jesse Newland
Rubyfuza2 HN recommenders · 2013–2021
32K views · talk · dated · Uploaded 2013-03-16
Jesse Newland describes how GitHub uses Hubot, its chat bot, to run operations such as deployment, monitoring, graphing and provisioning from chat rooms. He explains how this improves shared awareness and remote work culture.
The go-to reference for ChatOps, recommended for the idea of chat as a shared command line where teammates learn by watching.
38:47RailsConf 2014 - All the Little Things by Sandi Metz
Confreaks
40:20"Testing Distributed Systems w/ Deterministic Simulation" by Will Wilson
Strange Loop Conference
1:03:55🚀 TDD, Where Did It All Go Wrong (Ian Cooper)
DevTernity Conference
54:20Bret Victor - Inventing on Principle
Rui Oliveira
19:10Velocity NY 2013: Richard Cook, "Resilience In Complex Adaptive Systems"
O'Reilly
1:02:26Solving Problems the Clojure Way - Rafal Dittwald
Clojure/north
52:35Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017
GOTO Conferences
42:15"Performance Matters" by Emery Berger
Strange Loop Conference
6 comments from Hacker News
Absolutely. As I mentioned, we have 150+ rooms that we chill in, and most of our day-to-day work practices are tied into chat. Mention someone's name in a room they're not present in and we send push notifications to their phone and desktop- works great to pull people in as needed, without needing to distract them with the noise of being in the room 24/7. Our ops team is particularly in deep with chat. Instead of siloing everyone's process off individually in SSH sessions, many operations happen in chat, so everyone can learn and help out when diagnosing problems. They've built some really fascinating tooling around the problems they face- if you're interested, check out @jnewland's talk on ChatOps: [VIDEO]
There was also a talk titled "Chatops" about this specific thing if you want to learn more: [VIDEO]
There's a pretty good talk on "chatops" by the guys at GitHub[0]. Basically, it provides a dialog around a "shared command-line" where developers from learn from each other by simply watching chat. I think it's more beneficial to distributed companies. However, my question for those using chatops is basically how do you keep the repository for the bot "tame"? It needs access to _everything_ if you want it to be useful, which means having lots of secrets exposed to the bot. [0]: [VIDEO]
The DevOps approach: Just write a Slack plugin for all those things, if they don't already exist (they totally do tho). /s In all seriousness. Communication IS the baseline requirement for the super-app you speak of. The next requirement is making the platform talk to any data source. Slack does both of these things surprisingly well. Check this out: [VIDEO]
3. Never grand direct SSH to access the the staging/production environment. Automate the build and deployment environment on a system that developers don't have direct access. Instead of pushing changes, the bot pulls a specified release branch, builds, tests, and deploys the code. All without human interaction. If malicious code were somehow introduced from a developer's environment, it would be recorded and reflected in the commit history. AFAIK, that's how GitHub manages deployments via HubBot. See [VIDEO] . ----- To take things a step further, public-facing environments should be made immutable wherever possible. With the entire system being built and released as a whole. Docker alleviates some of the complexity and overhead but I think this space is where Unikernels have a lot of potential to shine. There's a very good talk about how the Wunderlist team used chaos and frequent destruction of their envronments to overcome fear and uncertainty here. [LINK] . Emphasis being placed on the the frequent disposal and recreation of environments rather than building long-running persistent environments. This setup probably won't work for long-lived systems (ex databases). In those cases, access via a transient environment like a USB bootable OS would be ideal to prevent persistent viruses/trojans. Ironically, the best current options are security-focused distros like KaliLinux that put a special emphasis on avoiding persistent state. Maybe one day soon we'll see an admin-focused
Github is well know for doing chatops for everything . In fact, I think they may have invented the term. Here's a relevant talk from 2013: [VIDEO]