kubernetes configuration drift minimization oppa gitops style

Over the past four decades, large-scale infrastructure consistency management people have continually re-discovered the inevitable entropy of "drift," wherein your pristine, uniform system configurations somehow get out-of-sync with reality.  Dozens of patterns, tools, and approaches have been tried and almost all are (still) running in large data centers to combat these misconfigurations. And, of course, new tools are emerging and older tools are adding more drift minimization features (e.g. salt) as they evolve.

Madhura Maskasky (co-founder of Platform9) has proposed an interesting approach using gitops and a role-based access control (RBAC) templating system to minimize kubernetes cluster drift.  Because kubernetes cycles pods continuously, and configuration deployment is exclusively from git, a significant amount of drift is reduced for "free," without adding another tool.

Drift minimization oppa gitops style.

Quis Custodiet Ipsos Custodes?

Here is an interesting take on detecting issues in the monitoring service and alerting the on-call person. Miedwar explained that their original method was simple and good enough: An AWS Lambda function periodically triggers and sends an HTTP health request to Grafana via proxy. When the health check fails, it triggers an incident in PagerDuty.  Elegant, independent, simple, pretty good. Why change?  It cannot see past failures between polling intervals. Their proxy is a single point of failure (SPOF).  Their new "trigger unless the system claims it is healthy" design resolves both problems and is just as simple.

For the Latin impaired, the title means "Who watches the watchmen?"

Camille Fournier explique 20 mesures d'un supérieure développeur de logiciels

Another great blog post from my buddy about measurements for differentiating  characteristics of a  senior software engineer.  The title of this entry is in french because the author's name is Camille Fournier.

another take on client retry logic

When clients have no visibility into global state (all clients' latencies & failure rates), there are simple heuristics we use for retry logic.  Marc Brooker introduces another approach (circuit breakers) that can be useful for certain statistical distributions of low client failure rates, especially under high server load.

Should we meet?

I saw this 20-second decision making tool (and flowchart) on my friend's blog and had to share.  Do we need a meeting?

Killing Titan by Greg Bear

The story line is starting to get more "way out there" with bigger mysteries and more-advanced (magical) awe-and-wonder technology.  The value and capabilities of the main characters comes into question, making it harder to suspend disbelief. 3/5 Stars.  I want to find out what happens next.

23 Years on Fire by Joel Shepherd

The series is starting to grow on me now that it's not all political and mysterious, advanced space aliens have appeared. The close combat has crossed the threshold into super-powers, which is disappointing.  Unless Ceephay Queen is available soon, I may continue the series. 3/5 Stars.

Oblivion (Starfire, book 8) by Steve White & Charles Gannon

It's a good ending to the series, tying up all the loose ends after a long, non-stop series of space battles.  I had to put the series down as it got too repetitive.  Despite the hackneyed overused tropes, the space opera is ok, 3/5 Stars.

Code Breaker buy Walter Isaacson

I think Brian Keating's Losing the Nobel Prize is slightly better. But this Jennifer Doudna story is pretty good.  I finished reading the book a few days before the patent office's final ruling on whose patent takes precedence for applications of CRISPR Cas9 for human gene editing (Doudna lost).  And I had no idea about the Biohackers who are editing their own genes.  The competition and races for her discoveries are fun, but the entire second half of the book, examining bio ethics, is a little boring.  It's still entertaining and fun, 5/5 Stars.

10 Principles for Growth as an Engineer

Today at work, someone pointed me at Dan Heller's blog post from a few years ago about how to be successful as an engineer.  They are:

1. Reason about business value
2. Unblock yourself
3. Take initiative
4. Improve your writing
5. Own your own project management
6. Own your education
7. Master your tools
8. Communicate proactively
9. Seek and exploit opportunities to collaborate
10. Be professional and reliable.

It's worth spending 3 minutes to read the whole article.

And the concepts are reminiscent of Elon's 5 Rules for Engineering.