What is the modern DevOps workflow?

I've have talked about the DevOps app stack and infrastructural building blocks utilized in modern app architectures, and the benefits of adopting these modern paradigms – even looking at how HashiCorp is focused upon different layers of the app stack. Let's complete the circle over those prior discussions, and take a look at what the basic steps of the modern DevOps workflow are, and what services and tools are trying to bridge the gaps – especially those trying to be an umbrella over it all, as modern DevOps platforms. While today's piece is more a technology lesson over the DevOps workflow, I'll weave in a few public companies (some quite new to the market), including GitLab, JFrog, Atlassian, HashiCorp, and Amplitude, as well as the still private Snyk – plus some mentions of heavily covered companies like Snowflake, CrowdStrike, Zscaler, and Datadog.

It's hard to find a company that isn't run by software, as the number of tools & services that enterprises rely upon continues to expand at a rapid pace. The companies that create all that software are the engine driving the App Economy (the rise of SaaS and mobile applications) and API Economy (for interconnecting services and data). The problem for these software development teams is that their workflow is typically managed in a piecemeal fashion, by stitching together multiple point solutions across a wide range of reliable and trusted self-managed or managed (SaaS) commercial or open-source tools. And while these tools typically have extensive integrations with each other (in order to do that stitching together), control, automation, and visibility over it all becomes more and more difficult as you add more tools.

The modern workflow for creating and deploying applications is shifting, with the entire software development lifecycle (SDLC) now being referred to as the DevOps lifecycle as the developer and operations sides merge more closely together. Enterprises seek to manage, secure, and monitor this lifecycle. And they now seek automation over it all, greatly reducing the amount of time that DevOps staff must waste on manually performing these tasks. Enterprises also seek to better tie this lifecycle into all of their business objectives and goals, to have the non-DevOps teams (product, sales & marketing) more tightly interwound with the app teams.

The DevOps Lifecycle

The stages for creating, deploying and managing an application are:

• Plan: Product and DevOps teams must plan out and prioritize the application feature set, and set and track progress over goals and milestones.
• Create: Developers are writing and locally testing application code, and must track their changes in a secure and collaborative way.
• Verify: Teams must test any code changes, and verify it meets an enterprise's standards. There are multiple layers of testing at different stages of the lifecycle, that businesses may or may not perform, or that may be done manually vs automated.
• Package: The application must be built and bundled with all the needed internal dependencies (libraries utilized), as well as the app stacks and infrastructure needed to run it upon. The final outputs are then kept in a package manager (artifact repository) in order to track and version the builds. These builds can be fed into later testing stages, and then, if approved, into release.
• Release: There must be a process for approving the final release across multiple changes to an application, and delivering (deploying) it into the final environment it will run in. These tools might automate across different deployment strategies to avoid any downtime, such as blue-green, canary, and shadow deploys [as discussed before in Datadog's platform dive]. As monoliths become more componentized and distributed (microservices or containers), the frequency of the release cycle of each individual one can drastically increase.
• Configure: The team must give the application the necessary configuration settings it needs – which might change for the environment it is running in, or the stage it is at in testing/release. The app also needs access to the credentials it uses for any external services it talks to.
• Protect: The enterprise must secure the running application in real-time, as well as the app stack and infrastructure it runs upon. This is typically been the domain of security providers, but DevOps tools are expanding into it heavily.
• Monitor: Watch the running application, as well as the app stack and infrastructure it runs upon. This is typically the domain of Observability providers [see that Datadog's platform dive again], but again, DevOps tools expanding into it. This also includes tools for watching the user behaviors within those applications, in order to extract better insights into that business's operations.

And over every stage within the entire DevOps lifecycle, you need:

• Manage: Enterprises need a uniform and automated process over all the steps, and complete visibility over the workflow processes.
• Secure: Security measures over all these DevOps workflow steps (including the code itself), to assure security & governance over the process. This has typically been the domain of DevOps tools, but security providers are beginning to expand into it as well.

There are a multitude of point solutions that provide a service or tool over one or more of these stages. However, a few companies, as well as the cloud providers themselves, are striving to become an all-in-one DevOps Platform to oversee it all, in order to manage, watch, and automate across every DevOps workflow to release app stacks (apps & the infrastructure it runs upon).

The DevOps Tools

So what tools apply to those stages? Just as there is a huge variety of programming languages and app frameworks, there is also a huge variety of tools.  It's such a mishmash (especially of open source point solutions) that there is a huge focus across these tools to the integrations between them. So the majority of DevOps platforms will work with all the other tools (even competing ones) for particular stages. Everyone wants to play nice with the others (but then provide an inline feature set that is comparable and they can upsell you on).  Here are the main tools used by a wide variety of app deployment use cases.

Workflow

• Team Planning and Tracking: Distributed enterprise teams need collaborative tools that provide team planning, documentation/wiki, issue tracking, and progress tracking. This includes tools like Atlassian Jira+Confluent, GitLab Plan, or GitHub Issues, as well as simpler workflow tracking tools like Monday.com and Asana.  [In my above graphic, I buried the "Plan" stage under the catch-all "Development" stage.]
• Developer Tooling: Developers are using tools for coding, local testing, & committing changes into a source code repository. These are typically handled via Integrated Developer Environments (IDEs) such as JetBrains's Intelli- series, Microsoft's VS Code (Visual Studio Code), or the open source Atom, and a wide variety of supporting tools for locally testing changes to apps and APIs. [These tools aren't really the focus of this piece, nor of my interest in this space.]
• Developer Setup: Developers typically are performing these steps locally on their laptop, which requires extensive setup in order to have all the various needed components installed in the dev environment (such as an app framework, web server, or underlying database). As mentioned before, it has become much easier to manage setup, as you can encapsulate the entire app stack in VMs and now containers. Tools like Docker and HashiCorp Vagrant are used to build easy-to-install developer environments.
• Source Code Tracking: Version control systems (VCS) – also known as source code mgmt (SCM) or repository – drive the rest of the DevOps lifecycle. A VCS tracks changes to code, and allows creating separate tracked lineages (branches) of app code for team collaboration and feature change tracking. Popular tools include GitHub (acquired by Microsoft), GitLab, and Atlassian BitBucket – all based on git, the overall winner of the battle between SCM formats. The cloud providers all have private git repo services as well.
• Testing: There are a wide variety of tools for manually performing testing, such as load/stress tests over apps and APIs, and browser automation tools that can be used for web app testing. Continuous Integration (CI) tools are services for continuously performing automated testing across the entire application, as code is committed to a VCS branch. It can perform automated unit testing per component, and, when combined with a CD tool, integration testing over the final build/bundle. Popular open source ones include Jenkins, TravisCI, and CircleCI.
• Deploying: Continuous Deployment (CD) tools are services that assure that after committed code is tested and approved, it automates the building, bundling, and deployment of the app. CI and CD services are now typically combined as an overall CI/CD pipeline tool, which allows for automating across the test, build, bundle, and deploy stages, and typically into release and distribution, to tie it all together over all deployment concerns. Beyond the open source ones above, popular solutions include GitHub Actions, JFrog Pipelines, and GitLab Pipelines. The cloud providers all have versions specific to themselves.
• Infrastucture Mgmt: Part of building and deploying an app might include needing to provision the underlying app stack or infrastructure it needs to run upon. As with testing, the notion of infrastructure ownership is also shifting left, to allow developers to control and own the underlying stacks their code runs upon. Provisioning tools, typically driven by Infrastructure as Code (IaC), allow for creating and maintaining the underlying infrastructure. Popular tools include HashiCorp Terraform and CloudFoundry, as well as Helm Charts in Kubernetes. The cloud providers all have IaC tools that cover over their services.
• Build Tracking: Package Managers, also known as Artifact Repositories, provide a centralized location where all final app builds are stored, versioned, and tracked. Container Repositories are a specialized version dedicated to just holding container images (templates & final builds). These tools provide the final build/bundles/containers to the rest of the lifecycle. Open source ones include Docker Registry and Harbor for Kubernetes, as well as managed services from JFrog Artifactory, Sonatype Nexus, GitHub Package, and GitLab Package.
• Secrets Mgmt & Proxies: DevOps teams must use a variety of Configuration & Secret Mgmt tools in order to set up the app configuration within each environment it is deployed into, and provide the secrets (like encryption keys) it needs, as well as the credentials it needs for any external services it intercommunicates with.  Beyond providing credentials, these tools are starting to become gateways/proxies, which allow them to provide just-in-time ephemeral connections (Zero Trust!) to external services. The cloud providers all provide secrets managers. Open source solutions include HashiCorp Vault+Terraform and CyberArk Conjur. PAM (privileged access mgmt) security providers are all moving into just-in-time services, including CyberArk, Centrify, and Thycotic, as well as dev tools like GitLab Config.
• Release Mgmt: Release & Deployment (or Distribution) tools help approve the final release of an app build or container, and then distribute it to the environments it needs to deploy the app into. It will typically tie into infrastructure provisioning tools to set up the cloud infrastructure needs for each deploy. These tools typically assist in tracking across a variety of deployment methods (like canary, blue/green, shadow deploy). Many of the CI/CD pipeline services are expanding into this, including GitLab Release and JFrog Distribution.
• Feature Set Mgmt: Extending past deployment, services exist to help companies introduce, deploy, and test new features within their applications, and to allow them to manage who can access the changes. Feature Flags are helpful to introduce new features to A/B testing with a subset of users, or to users within a specific geo-region. Example services include LaunchDarkly and Split.io.
• Managing Distributed App Stacks: There are additional tools for particular app stacks, which have been covered before. Container orchestration, such as Kubernetes or HashiCorp Nomad is needed for managing the various components of an app as one cohesive app stack. As previously discussed, distributed application architectures like microservices and serverless functions need tools such as service mesh to securely intercommunicate.  The cloud providers have specific ones, as well as HashiCorp Consul, which allows app stacks to span environments in hybrid and multi-cloud.

Automation is the critical element across these DevOps tools and services, which typically fall to the CI/CD pipeline tools. They must typically tie into source code repositories, Infrastructure as Code (IaC) provisioning tools, package managers, and container repositories, and have (or tie into) extensive security and monitoring tools.

• ChatOps-based automated workflows are driven by chat/notification systems, such as Slack and Mattermost. DevOps staff are able to trigger actions to enable bots to run portions of the workflow, such as running a CI test, or deploying to production.
• GitOps-based workflows are driven by the VCS system's commit & branching mechanisms. It was coined by Weaveworks in 2017, then updated in 2018. This leaves VCS systems, such as GitHub and GitLab, as the central driver of the workflow, serving as the source of truth that drives all the other stages.

Most pipelines want to be the centralized hub of automations across the other tools utilized, typically via scripting ("pipeline as code"). Some, such as GitLab Pipelines, are taking it a step further, by becoming a pipeline of pipelines via a no-code interface, in order to stitch together existing CI/CD & infrastructure provisioning tools across open-source, SaaS, or cloud providers.

Beyond the workflow-focused tools, let's break out the security and monitoring capabilities needed over all deployed applications and the underlying infrastructure.

Security

Security tools exist that are starting to blend DevOps into DevSecOps, and, like testing and infrastructure mgmt, drive how security is shifting left to DevOps teams – leaving security teams to more focus on the holistic "big picture" instead of individual components.

Pre-Release Security

Several features exist to boost the security of an app before it is released, which can be automated in the CI/testing stage:

• Code quality can be monitored, by tying linting (static code analyzers) and code quality tools into the VCS commit or CI process. These help assure the developer's committed code adheres to that enterprise's policies and best practices.
• Static Application Security Testing (SAST), aka code scanning, looks at source code and binaries to scan for known vulnerabilities.
• Secrets scanning extends that code scanning further, by assuring no credentials are being exposed in code. That critical info should be being handled by a separate Secrets Mgmt system that the app should tie in to.
• Software Composition Analysis (SCA), aka dependency scanning, identifies and looks through the libraries an app internally uses, to track known vulnerabilities in the underlying open-source libraries being utilized.
• Container scanning does that same type of thing, but instead scans a container repository to check the underlying OS and software releases in the pre-built container images.  

Modern DevOps platforms are striving to do all the above. GitLab Secure, JFrog Xray, and Snyk have a wide variety of tools for use in pre-release security over DevOps workflows. Security platforms, like Palo Alto Prisma Cloud (through their acquisition of Twistlock) and CrowdStrike, are starting to move into container vuln scanning as well.

Post-Release Security

After release, the security features start blending between what DevOps tools provide for real-time testing, as well as services from cloud workload & networking security providers.

• Dynamic Application Security Testing (DAST) tools allow DevSecOps teams to do passive (traffic scanning) and active (simulating malicious attacks) against apps and APIs, in a test or production environment.
• Interactive Application Security Testing (IAST) tools combine the SAST and DAST features into an embedded agent within the app code.
• Cloud Security Posture Mgmt (CSPM) are specialized scanning tools that look at cloud containers and VMs for misconfigurations & policy violations. This includes [well discussed] solutions from CrowdStrike, Zscaler, Palo Alto, and Datadog.
• Cloud Identity & Entitlement Mgmt (CIEM) tools are helping to manage the identity and access allowed within apps. IAM/PAM tools like CyberArk are entering this heavily, as is Zscaler after its acquisition of TrustDome.  [CSPM and CIEM features were discussed before in my Zscaler deep dive.]
• Web Application and API Protection (WAAP) allow for firewall protection over apps and services, to help control the network access of incoming requests into the app. This includes next-gen WAF (web app firewall) solutions from Cloudflare, Fastly, Palo Alto, and Foritnet.
• Cloud Workload Protection (CWP) tools are real-time (runtime) security features that help protect the underlying VMs, containers, and Kubernetes, as well as the app traffic between them. this includes CWP solutions from CrowdStrike, SentinelOne, Zscaler, Datadog, and Palo Alto. [I'll be diving into this area more.]
• Runtime Application Self-Protection (RASP) is a newly emerging method of embedding security and behavioral analytical capabilities directly into applications, in order to make real-time decisions about incoming requests. WAAP providers, like Signal Sciences (Fastly) are starting to provide these features, as are security-focused observability companies like Datadog and Dynatrace [as discussed before with Datadog in premium].

Monitoring

DevOps need to watch over their apps, app stacks, and infrastructure, by watching all logs, metrics, and traces, as well as over the automated CI/CD pipelines themselves. Open-source enterprise-level Observability can be done via Elastic or Sentry, and a number of SaaS services like Datadog, Dynatrace, New Relic, and Splunk. These observability tools are typically tied into other Incident Response tools for notifying DevOps teams when incidents in app stacks and infrastructure arise, as well as Service Desk ticketing systems in order to manage requests being made to an Ops support team. Systems like PagerDuty and ServiceNow are striving to do both.

Beyond infrastructure metrics and app logs, Application Performance Monitoring (APM) has huge benefits for DevOps teams. From my Expanding Observability [premium] post in Aug-21, APM tools can provide:

• Service mapping allows for creating a visual map of how various distributed components intercommunicate within an app architecture.
• Distributed tracing interconnects the various distributed backend components hit in a request.
• Code profiling can show requests down to the code level, to see how code changes are affecting things, and what functions or lines are bottlenecks.
• Database profiling can show how database queries are performing (akin to code profiling, but within the database engine).
• Digital Experience Monitoring (DEM), also called Real User Monitoring (RUM), allows for tracing the requests and responses of a single user, to see what app components they hit and what type of performance that user sees. Features like Session Replay have emerged in DEM/RUM tools, that allow developers and product teams to play back a user's actions within an app, in order to see through their eyes.
• Synthetic testing allows for test runners to be used for the continuous testing of application releases, and can help automate various deployment strategies (like blue/green, canary, A/B testing, etc), as well as tie more closely into DEM to trace synthetic users across test stacks.

Open-source distributed tracing can be done via Jaeger or Zipkin, while new SaaS point solutions have emerged, like Honeycomb.io for observing microservices, and LogRocket for front-end performance and session replay (combined with product analytics).

Productizing the Application

Besides viewing the real-time metrics and logs being generated by an app stack, the business needs to track how users are using the application, and the behaviors they are taking through the app components and UI. In particular, these types of in-app metrics must be tied back to business goals and KPIs. This starts moving the focus of these tools from DevOps observability to allowing product teams to measure and react to how the user is utilizing the app, and conduct A/B tests against new features.

Observability APM tools (like Datadog, Dynatrace, and New Relic) can be used to mix business intelligence-related metrics into telemetry data. But a new trend has emerged to instrument your application into a Product Analytics (PA) tracking service, in order to constantly monitor user behaviors in how they utilize your app's features. These typically allow teams to quickly adjust the app towards business KPIs and goals. Open source enterprise-level tools like Snowplow and GitLab Monitor are helpful for user behavior tracking & analytics, and other SaaS point solutions are emerging in this space, including Amplitude, Heap, Pendo, and Mixpanel. Observability companies are starting to move more into BI features as well, and LogRocket is one SaaS service bringing the gap between observability and product analytics. I expect other Observability companies to bridge this gap, and become more BI/Product focused.

Data warehouses, such as Snowflake, exist to have a view over the entirety of a business, and to allow the insights gained to spread from product teams to sales and marketing. These product analytic services typically integrate further into data warehouses, to then fully inform the entire organization (as Heap, Pendo, and Mixpanel allow for). Heap won Snowflake's Data Marketplace Partner of the Year in Jun-21, and Mixpanel is a featured partner as well. Amplitude announced a new partnership and integration with Snowflake in Nov-21.

A Complete Ecosystem

Let's redo my diagram above, to focus on the major areas across the app lifecycle. There are a wide variety of tools addressing the DevOps workflows, addressing either the developer/team needs, the CI/CD pipeline, the release distribution, or the monitoring & security stacks.  

Many managed services (all coming from different starting points in this ecosystem) are starting to move away from just being separate integrated point products, and more into becoming a fully managed & automated service over it all. It's a return to the bundled vs unbundled product strategies. Right now, all these tools are mostly unbundled point solutions over specific stages of the workflow, but several of these platforms (especially coming from the CI/CD pipeline automation tools) are moving into bundled solutions – an overall DevOps Platform that can manage, see, secure, and automate over it all.

Weaveworks, Harness, & Cloudbees are all private SaaS services providing this type of end-to-end software delivery, while public companies GitLab, Atlassian, and JFrog are all striving to become bigger and bigger players across these DevOps workflows (in very different ways). As they flesh out their platforms more fully, they want to become the sole platform for DevOps workflows, eliminating the various point solutions they interconnect with. This likely provides a lot of upsell potential, if they are successful.

• GitLab is an enterprise open source company moving beyond its roots in VCS and CI/CD pipelining control, to try to provide a complete DevOps platform over all the workflow stages (Plan, Create, Verify, Secure, Package, Release, Configure, Monitor, Protect, and Manage). They have a roadmap to "continue rapidly maturing our DevOps platform so that GitLab is able to replace any DevOps point solution". Interestingly, beyond being app-focused, they also see how their vision will align with tracking and managing ETL and analytical processes as well.
• JFrog is also making heavy moves to be a more complete and automated DevOps platform, from its roots in package & release mgmt (Artifactory). It has added security capabilities with Xray, CI/CD pipeline capabilities in Pipelines, release mgmt in Distribution, and oversight over it all in Mission Control. It seems heavily focused on being the secure & automated platform that serves as the glue between all the different on-prem and cloud environments, and is moving extensively into the IoT/OT side as well with some recent acquisitions (as are the cloud providers).
• Atlassian, as always, is a major player in the DevOps space, with its issue tracking and wiki core, plus Bitbucket for VCS and Bamboo for CI/CD, as well its developer tools of Crucible (peer code review) and Fisheye (source code search). It continues to add more planning and governance features, and announced a new Open DevOps initiative in Apr-21, where it seems to want Jira as the centralized driver of workflows instead of git ("TicketOps"?). GitLab is a major partner in this new initiative. However, Atlassian also remains busy moving heavily to Service Desk and Incident Response, as well as generalizing its platform to handle any type of workflow (not just DevOps) in Jira Work Mgmt and Trello.

Moving more upmarket with more enterprise-focused features, Value Stream Management (VSM) is the new industry term for the management, visualization, and optimization of the entire DevOps lifecycle across a large enterprise. These platforms tie the workflows and apps more directly into business goals, plus add a heavier emphasis on governance and compliance across the workflows and automation. Platforms like Digital.ai, ServiceNow, and Plutora are heavily focused on the largest enterprises. These all seem more top-down (selected by the CIO) than traditional bottom-up DevOps tools.

Add'l Reading

As mentioned last post, I released a HashiCorp deep dive earlier this year that explains their product line (Consul, Vault, Terraform, et al). Peter Offringa of Software Stack Investing did a GitLab deep dive in July. For a better understanding of DevOps workflows, Technically has helpful pieces explaining DevOps workflows, continuous integration (CI) tools (premium), and what HashiCorp does (premium).  

I explore hypergrowth companies within the DevOps, edge network, data & analytics, and cybersecurity. This was originally a post from my premium service from last December. Sign up for Premium if you want weekly insights into hypergrowth tech companies and where they are going in their platforms and market pivots. I currently have in-depth research into observability, DevOps tools, data & analytics, edge networks, and cybersecurity (Zero Trust, SASE, XDR, CNAPP) industries, including heavy coverage of CrowdStrike, SentinelOne, Zscaler, Cloudflare, Snowflake, and Datadog currently.  

I have gone even further in DevOps in premium, including how cloud providers are expanding out into regional data centers & IoT micro-environments for containerized application deployment, and how edge networks like Cloudflare are creating new global-spanning serverless dev environments and a new breed of edge-native primitives for dev use. (With more coming this week in their Birthday Week!)

- muji