HHHYPERGROWTH

Clarity from Elastic

muji -

I attended an Elastic{ON} virtual event, and while there was not much I already didn't know, it really CLARIFIED things. I recap the talks and Q&A below. You can no longer call their product lines sprawling, as they seem very focused right now. I like watching these events for the product news, and what I found especially interesting this time was the Q&A going on during presentations in a side text-based chat room, where there were some pretty interesting tidbits (even from the CEO).

It was nowhere as slick a virtual experience as Oktane20 was, Okta's customer event I recently watched. This was just one long Zoom meeting that flipped from presenter to presenter. CEO Shay Bannon was pretty unpolished on this - or maybe it was how he was backlit by an outdoor window, making him washed out. But he set the tone in his keynote, walking through the focused pillars and the focus & products within each line.

Platform Core

I really really like how they have boiled down what they do into very clear paths. It always seems like Elastic is moving in a lot of directions, but this boils into the 3 distinct markets.

The core is in the middle, which I've covered before. The Elastic Stack (nee ELK stack) consists of Elasticsearch (search database), Kibana (visualization, dashboard and ML interface), Beats (open-ended ingestion interface), and Logstash (file conversion). This is available as the open source or enterprise supported release, in order to run self-managed instances on whatever infrastructure desired.

At the bottom is their managed hosting methods for the stack beyond self-managed:

  • Elastic Cloud (cloud SaaS provider, billed on usage)
  • Elastic Cloud Enterprise (managed SaaS deployed on-prem)
  • Elastic Cloud on Kubernetes (container cluster manager)

Product Lines

On the top, we have the 3 distinct verticals they have built over this core & methods of use. These pillars are leveraging the core Elastic Stack into multiple directions:

  • Elastic Enterprise Search
  • Elastic Observability
  • Elastic Security

This direction came about through their acquisition of SwiftType, a SaaS private that built these services over Elasticsearch open-source.

Competition: MongoDB Atlas Search, Algolia, Coveo

Products:

  • Workplace Search = Search engine & interface over a internal content and SaaS tools (email, Slack, Github, Google Drive, Office 365).
    Example user: A company's workforce, as a search engine over internal content, including each user's private content like emails and chats.
  • App Search = Search engine & interface over provided content, via their API (for e-commerce, or inline search within SaaS or mobile apps).
    Example user: A SaaS provider's customers or mobile app users.
  • Site Search = Search engine & interface over web content (for publishing, blogs, static written content).
    Example user: A company's customers or mobile app users.

Elastic Observability

Allows a centralized search capabilities & interface over all your infrastructure & services (SaaS services, APIs, web apps, or mobile apps).

Competition: Datadog, Dynatrace, NewRelic

Search over these types of internal data:

  • Metrics
  • Log files
  • App performance monitoring (APM)
  • Uptime

Unified ML over all that data, to allow "AIOps" features.

Example user: A company's IT & developer teams, as a search engine and visual dashboard over internal infrastructure and services.

Elastic Security

It started as as Observability, but their stack pivots easily into security concerns over that same data, providing monitoring of a different kind.

  • SIEM = Cybersecurity search engine and visual dashboard over that Observability data, but geared for a different audience and purpose -- the detection & tracking of intrusions and cyber attacks. Elastic provides SIEM-specific dashboards in Kibana, to provide a security-centric view into same observability data that IT uses. The initial GA of SIEM has released, for observability & monitoring of a company's network -- but Elastic is still iterating. They have an expanding list of pre-build collectors for a wide variety of security equipment & data sources. New partnership & integrations announced with Palo Alto for their SOAR capabilities (orchestration of incident response).
    Example user: A company's Cyber & IT teams, as a search engine and visual dashboard over cybersecurity & intrusion detection, over that same internal infrastructure and services data from Observability.

    Competition: Splunk, LogRhythm
  • Endpoint = EPP from their acquisition of Endgame, an installed app to protect a user's devices from malware and intrusion. Has been directly integrated into Elastic SIEM.
    Example user: A company's IT team, as a way to protect & manage workforce devices and BYOD personal devices.

    Competition: CrowdStrike, Microsoft, Blackberry/Cylance, VMWare/Carbon Black

Hosted Services

Number of global cloud data centers is growing across the big 3: AWS, Azure, GCP.

No longer need to self manage, can config, manage, scale easily. High Availability across 2-3 availability zones, if desired. Comes with tech support & uptime SLA. Data encrypted at rest and in transit, and has complete tenant isolation (they are in separate containers).

FedRAMP still "in process", ETA this summer. They are offering a free 30d trial to sample it in AWS GovCloud (US - East), which has extended E2E encryption over regular Elastic Cloud.

Deployment templates in Elastic Cloud: I/O optimized, compute optimized, memory optimized, hot-warm, app search, cross cluster search.

Security

Palo Alto has new partnership w/ Elastic, and is co-sponsor of the conference. They announced a new product, Cortex XSOAR.

  • Next evolution of product line coming out of their Demisto acquisition and its SOAR product, now rebranded as "XSOAR" (next-gen security automation & response).
  • Does orchestration, automation & case mgmt for security ops & incident response.
  • Has workflow automation engine (flow chart UI) over 100s of security products and 1000s of actions.
  • Security ticketing system & collaboration platform.
  • Ties directly into Elastic SIEM.

And now ... the even MORE interesting stuff

Here are the Q&A of interest, that were asked during the webinar. CEO Shay Banon was answering many of these (in a text chatroom) after his keynote.

Elastic Security

  • So is EPP a unified solution thru the UI or separate from the Endgame solution?
    Elastic: Today, we have achieved API level integration between Elastic Stack and Endgame console. From the stack, customers are able to triage and analyze Endpoint data and alerts then leverage the API integration to pivot into the Endgame console for further analysis and response actions. We have an aggressive road-map to collapse the Endgame console into the stack and fully integrate its capability creating a unified UI.
  • Is there a tentative date for the Elastic/Endgame integration?
    Shay Banon: Not yet, hopefully measured in months, but it will be iterative, so first release will not have all the Endgame features. We will support and continue to enhance running Endgame platform running next to the Elastic Stack until parity is achieved. [I believe this will be a way better product in the next 6-12 mo; they iterate fast.]
  • Will Endgame in ELK be in open source or enterprise package?
    Shay Banon: We are still looking into it, which part will be free and which parts will be in subscription lanes, more to come once the integration matures. [It is only in their Platinum top enterprise package right now, but seems to be hints that they are still feeling it out...]
  • What data sources can you pull into this tool?
    Elastic: The main requirement for SIEM data is that it adheres to the Elastic Common Schema (ECS). The easy button for that is Beats, which can pull from pretty much everything you have, but you can transform data from any source into ECS format using Logstash, Elasticsearch ingest node pipelines, or your own tools.
  • Do you have Elastic Common Schema (ECS) conversions for known security applications/formats?
    Elastic: Yes. We have many different integrations built into Beats, called Modules. For example, Filebeat has integrations with Suricata, Zeek/Bro, and PAN and Cisco firewalls. We have a team dedicated to building these out! The full list for Filebeat is here.
  • Can you explain how Elastic Security work with threat intelligence data?
    Elastic: Out of the box, Elastic research team develops advanced trait craft analytics mapped to MITRE ATT&CK to detect and stop adversarial activity. Additionally we provide our customers the ability to create their own analytics to detect and prevent against custom threats. Users can upload IOC via the Endpoint UI and exposed API to search/hunt for anomalous activity across the enterprise.
  • How often do you guys put out new rules?
    Elastic: New rules are added for each release. The last release included the first 92 Detection Rules, the release prior added 30+ ML rules. We are on a DevOps release cycle and historically version updates occur every 6-10 weeks. Also, many users create their own rules specific to their environment and share these rules between sister organizations. The out-of-the-box rules are focused on securing from known attacks in the MITRE Att&CK Matrix as well as other commonly evaluated security rules. [They release at lot, so will be fairly up to date at all times. But compare this model to CrowdStrike, who can react in an instant globally across their entire customer base. I continue to insist that cloud-based EPP providers like CRWD will ALWAYS be more nimble, and hence BETTER, than Elastic Endpoint because of this massive difference in reaction speed (weeks vs seconds) and scope (global view over all customers vs a single customer's island).]
  • Will Elastic integrate some type of open source intel?
    Elastic: Open Source Intel (OSINT) is some of the most disparate and unstructured data out there! It's also huge and really utilizes Elastic's strength in scaling horizontally. We've provided tools to help here since the earliest of days at Elastic (remember the Twitter Logstash plugin?). I've seen customer environments where Elastic's search, security, and observability solutions incorporate open source intel but very much in a bring your own data style. Barclay has spoken about correlating social feeds with their Observability feeds to measure whether they are spotting problems before the customers do! In terms of product roadmap, we don't have plans to cook OSINT sources directly, but we are open to ideas if you think there is something generic and broadly useful to the community.

FedRAMP Certification

[This is important as it opens the floodgates for Federal use of Elastic SaaS services. Elastic is widely in use now across Fed govt. This opens up a massive market for the managed cloud service.]

  • Is there any plans to get certified for FedRAMP High?
    Shay Banon: We are first focused on Medium, and then look into High. We do provide downloadable products (including orchestration support ones) so if there are additional needs, you can run it yourself on whichever infrastrucure and security constraints you might have.
  • Will you pursue IL-5/IL-6 level certifications? [These are higher level DOD Impact Levels above FedRAMP High - IL-6 is "classified secret".]
    Elastic: Heck yes, discussions have definitely started on this but it's later in the road-map for us! If you'd like to talk through how to use our tools in an IL-5/IL-6 project reach out and let's walk you through options between now and Elastic offering a hosted platform at that level.
  • On your product road-map [on Enterprise Search], are you planning a connector for: SharePoint, Office 365, Microsoft Teams?
    Elastic: SharePoint and Office 365 are supported today, and Microsoft Teams will be added in the future.
  • Are the workplace search connectors open source? Our team has a need to customize the sharepoint connector and instead of rewriting one from scratch it would be great to tweak the existing one that already works great.
    Elastic: Connectors are not open-source today. Writing a connector for SharePoint using the Custom Source API is the most practical and efficient way to achieve your goal today. Custom Sources support all the features you will need: flexible schema, customizable result display, document-level permission support and more.
  • Will Workplace Search be an open source part of Elastic Stack or a paid product?
    Elastic: The Workplace Search offering will be part of our Platinum Subscription. If you do not have access to a Platinum license, a trial license is automatically applied for you upon the first successful boot. Trial licenses are valid for 30 days. It's currently available as beta software, in the form of a self-managed deployable binary. It can be downloaded free for experimentation and evaluation.
  • How do you get Elastic Workspace to interface with Sharepoint in Office365? Is there a connector that gets setup so Elastic can crawl it?
    Elastic: Sharepoint Online has a connector up and documented as part of the current pre-GA Workplace search. Office 365 is also planned. On-prem Sharepoint and other behind the firewall data sources are currently road-map and the product can load them through the the Custom API Source for now. There are a bunch of open source crawlers available, but there is no on-prem crawler at the moment.
  • Can a workplace data source be documents on your file system?
    Elastic: Yes! Workplace Search allows for the ability to bring in your data with the Custom API Source. As long as you have a crawler/connector that walks through the docs on your file system, you can push output the data to the Workplace Search Custom API.
  • Is workplace search designed to be integrated into our existing sites look and feel?
    Elastic: Yes! We're providing Search APIs so you can embed the search experience into any other platform you're already using. So if your organization uses Confluence, and you have users that live in that search box, you can keep using that interface and still see all your other sources since we provide the integrations to search everything from there.

Miscellaneous

  • In general, how long does an application stack have to be running before useful machine learning data has been collected?
    Elastic: To some extent “it depends” based on your data. As a general rule the unsupervised machine learning jobs need three periods to learn periodicity, and about three seasons to learn seasonality. An example would be monitoring user logins in an office that works 9-5 each day (periods) M-F (seasons). It should take a few days to learn normal daily behavior, and a few weeks to learn normal weekly behavior.
  • With reference to the observability context, is there an out of the box functionality that takes the results from the Prometheus exporter and exposes that to Kibana for various visualization dashboards? So effectively replacing the functionality that Grafana provides?
    Elastic: Yes. We have put a lot of work into enabling Prometheus users to bring their data into the Elastic Stack without sacrificing anything. You can read more about it here. Also, stay tuned for our next release, 7.7, which will add additional features for Prometheus users. [This is good, as Prometheus metrics format are heavily used in containers, as it is the one real-time metrics feature available directly within Docker. Containers are a huge tech wave on the software development front right now, gaining a lot of momentum as it really helps how software stacks are created, deployed and scaled. Of course, Beats also has native Docker & Kubernetes modules for container observability. ]["Prometheus", the open source software, is a bit confusing at first glance -- it's both a metrics data standard, as well as an open-source server product. Elastic Stack would be a competitor to the server product. Prometheus is addressing this by separating its efforts, and the Open Metrics standard is splitting out.]
  • What are you doing with CMMC?
    Elastic: Cybersecurity Maturity Model Certification (CMMC) is a new DOD security integration standard that all DOD contractors must be certified within. Elastic is working with some integration partners that are working towards certification and can help customers with this. [I wasn't familiar with CMMC. All DOD Contractors are being required to obtain it this certification. More here on it.]

- muji