For Developers, by Developers


[] Monitoring in the time of Cloud Native

The infrastructure space is in the midst of a paradigm-shifting change. The way organizations — from the smallest of startups to established companies — build and operate systems has evolved. Containers, Kubernetes, microservices, service meshes, immutable infrastructure are all incredibly promising ideas which fundamentally change the way we operate software. As more and more organizations move toward these paradigms, the systems we build have become more distributed and in the case of containerization, more ephemeral.

[] Docker Platform and Moby Project add Kubernetes

Today we’re announcing that the Docker platform is integrating support for Kubernetes so that Docker customers and developers have the option to use both Kubernetes and Swarm to orchestrate container workloads.

[] Reasons Kubernetes is cool

I will try to explain some reason I think Kubernetes is interesting without using the words “cloud native”, “orchestration”, “container”, or any Kubernetes-specific terminology :). I’m going to explain this mostly from the perspective of a kubernetes operator / infrastructure engineer, since my job right now is to set up Kubernetes and make it work well. I’m not going to try to address the question of “should you use kubernetes for your production systems?” at all, that is a very complicated question. (not least because “in production” has totally different requirements depending on what you’re doing).

[] Health Score Metrics as a Software Craftsmanship Enabler

The notion of software craftsmanship is sometimes a muddy one. On the one hand, engineers find it hard to grasp and materialize craftsmanship, which is an abstract objective that, by itself, provides little guidance to the software engineering practice. On the other hand, craftsmanship is often narrowed down to a handful of "best practices" that engineers are expected to follow. Neither of these limited definitions helps much in improving software quality. During 2016, an R&D initiative for software craftsmanship was one of the technical priorities across the engineering organization at LinkedIn. As a part of this initiative, we decided to take the approach of quantifying some components of software craftsmanship to help guide our engineers towards better software development practices and software quality. The ultimate goal is to provide concrete and actionable guidelines for every piece of software being created at LinkedIn. With this goal in mind, we built a health score platform that collects and presents craftsmanship-elevating metrics, and provides a framework for convenient extension. We also proposed and implemented an initial set of metrics and plugged them into this platform.

On the design of distributed programming models

On the design of distributed programming models Meiklejohn, arXiv 2017. Today's choice is a lovely thought piece by Christopher Meiklejohn, making the case for distributed programming models. We've witnessed a progression in data structures from sequential (non-thread safe) to concurrent, to distributed (think CRDTs). Will the same thing happen with our programming models? 

[] Tools and Practices for Documenting Microservices

The architectural pattern has gained popularity over the past years, and although not everyone is completely sure what “doing it right” looks like, it’s a concept that suits modern needs and is here to stay for the foreseeable future. Over the past month, multiple people asked me about what tools and practices I recommend for documenting microservices and application architectures that use the pattern.

[] Logging with Kubernetes and Humio

Kubernetes is an interesting problem when it comes to logging. With all those containers created and destroyed, logs become the only dependable window into what’s happening, but working with them becomes significantly more complex. Humio is all about getting straight to the most important detail in your logs, especially when those logs are generated in huge volumes. That’s why we’ve created a integration between Humio and Kubernetes: kubernetes2humio.

[] Multiple Perspectives On Technical Problems and Solutions

Over the years, a number of people have asked about the details surrounding Etsy’s architecture review process. In this post, I’d like to focus on the architecture review working group’s role in facilitating dialogue about technology decision-making. 

[] Publishing with Apache Kafka at The New York Times

At The New York Times we have a number of different systems that are used for producing content. We have several Content Management Systems, and we use third-party data and wire stories. Furthermore, given 161 years of journalism and 21 years of publishing content online, we have huge archives of content that still need to be available online, that need to be searchable, and that generally need to be available to different services and applications. On the other side we have a wide range of services and applications that need access to this published content — there are search engines, personalization services, feed generators, as well as all the different front-end applications, like the website and the native apps. Whenever an asset is published, it should be made available to all these systems with very low latency — this is news, after all — and without data loss. This article describes a new approach we developed to solving this problem, based on a log-based architecture powered by Apache KafkaTM. We call it the Publishing Pipeline. The focus of the article will be on back-end systems. Specifically, we will cover how Kafka is used for storing all the articles ever published by The New York Times, and how Kafka and the Streams API is used to feed published content in real-time to the various applications and systems that make it available to our readers. 

[] Resources for Getting Started with Distributed Systems

I’m often asked how to get started with Distributed Systems, so this post documents my path and some of the resources I found most helpful. It is by no means meant to be an exhaustive list. It is worth noting that I am not classically trained in Distributed Systems. I am mostly self taught via independent study and on the job experience. I do have a B.S. in Computer Science from Cornell, but focused mostly on graphics and security in my specialization classes. My love of Distributed Systems and education in it came once I entered industry. The moral of this story is that understanding distributed systems doesn’t require academic intervention to learn and excel at.

1 2 3 4 5 6 7 8 9 10 11