Home » Cybersecurity » Events » QueryCon 2019: A Turning Point for osquery

QueryCon 2019: A Turning Point for osquery

by Lauren Pearl on September 20, 2019

Has it really been 3 months since Trail of Bits hosted QueryCon? We’ve had such a busy and productive summer that we nearly forgot to go back and reflect on the success of this event!

On June 20-21, Trail of Bits partnered with Kolide and Carbon Back to host the 2nd annual QueryCon, at the Convene Old Slip Convention Center in downtown New York. We beat last year’s attendance with 150 attendees from around the globe. The 14 speakers presented talks on osquery ranging from technical presentations on Linux security event monitoring to discussions of end-user research. We saw familiar faces from last year’s event in San Francisco, and we met many new teams interested in osquery.

Tania McCormack of Carbon Black presented her user research on introducing osquery to new audiences.

Last year’s inaugural QueryCon brought us all together in person for the first time. QueryCon 2019 strengthened our sense of community and proved a catalyst for positive change: Our productive collaboration generated community-based and technical changes that have put this project back on track.

Sponsorships Available

A new foundation

On June 18th, the day before QueryCon, the Linux Foundation officially announced that they would be taking over ownership of osquery from Facebook. Under the Linux Foundation, the new osquery Foundation will be directed by a Technical Steering Committee (TSC) consisting of engineers and developers from Facebook, Trail of Bits, Google, and Kolide—companies that are using osquery and have committed to supporting the project. The TSC members are:

Teddy Reed (Facebook)
Alessandro Gario (Trail of Bits)
Zachary Wasserman (independent consultant)
Victor Vrantchan (from Google, but working independently)
Joseph Sokol-Margolis (Kolide)

This change was exciting news to a growing list of companies who rely on osquery for endpoint protection. As we reported in April, osquery outgrew its original home as a Facebook project, and its community’s expectations and needs now exceed what Facebook can be expected to manage on its own. A new community-based governance model was needed, and conference attendees were eager to discuss the change. We hosted a panel discussion with Facebook’s lead osquery maintainer, Teddy Reed, and representatives from the new osquery TSC.

Facebook’s Teddy Reed led a panel discussion and Q&A with members of the osquery community about plans to transfer stewardship of osquery from Facebook to an open-source foundation.

How the foundation will work

The Linux Foundation functions as a steward for osquery, providing various funding and management platforms. (Learn more about their stewardship model here.) The new osquery TSC will guide and maintain the project with the help of contributions from the greater community, and Trail of Bits will commit to biweekly office hours for public comment and transparent project governance.

Meanwhile, Facebook will turn over credentials and control of funding, infrastructure, hosting, and engineer review to a new committee of maintainers (of which Facebook will remain a member). The organizations on the TSC are contributing significant engineering time to establish build and release processes, and a forthcoming funding platform on CommunityBridge will allow sponsorship.

Technical decisions

The TSC has a significant backlog of contributions to work through, but we’re already seeing a massive acceleration of activity on the project.

First, osquery core will be updated to feature parity with osql, the community- oriented osquery fork by Trail of Bits. The initial goal is a monthly release, with alternating “developer” and “stable” releases. Another big priority is to merge all major independent efforts and private forks into a single canonical osquery that everyone can benefit from.
Once Trail of Bits resolves the technical debt that has accrued on the project—build toolchains, dependency management, CI systems—it will maintain these components and focus on client-driven engineering requests for osquery. Other stakeholders are also contributing a backlog of Pull Requests, which will be prioritized and merged as soon as possible.

A proliferation of committed PRs

One way to track the health and activity of a project on GitHub is by Pull Requests. Over nine months, from September 2018 to the day before QueryCon, there were roughly 35 PRs merged to the osquery project, with only a few from the community outside Facebook. In just the 12 weeks since QueryCon, nearly 90 PRs were successfully merged (representing about 113 commits). More importantly, the majority of those contributions were from outside Facebook.

Trail of Bits alone is responsible for approximately 44 of the PRs merged this summer.

Some highlights from our recent contributions:

#5604 and #5610: The new osquery Foundation version of osquery was kicked off by merging the Facebook and Trail of Bits versions of osquery. This meant we restored CMake build support and set up the public CI, which were key improvements brought over from the osql fork.
#5706: We refactored the build scripts so that all of osquery’s third-party library dependencies will build from source on Linux. This absolves the need for the project to host and distribute the correct versions of pre-built binaries for these libraries (a job that previously relied on Facebook); improves compatibility across different varieties of Linux; is a prerequisite for our goals of reproducible builds and offline builds; and, finally, avoids incompatibilities arising from system-installed versions of these dependencies.
#5696, #5697, and #5640: We fixed and vastly improved the table for querying the certificates stores on Windows. It is now possible to use osquery to list all per-user certificates, whether they are in a registry hive or on the filesystem, and whether or not those users are even logged in at the time of the query. Searching your fleet for anomalous certificates is an important security monitoring and incident response capability, as it can be an indicator of compromise.
#5665: We fixed several bugs that we found with the use of dynamic analysis (fuzzing and Clang’s Address Sanitizer). Soon, we plan to incorporate both static and dynamic analysis passes into the CI system, for proactive detection of code vulnerabilities in osquery. This is a best practice that Trail of Bits recommends for all of our clients, and we’re happy to contribute it to the security of osquery.

A new stable release

During a community workshop at the end of the conference, osquery users and TSC members discussed the best path to the next stable release.

Prior to QueryCon 2019, the most recent major cross-platform release was August 2018. Seven days after the conference, Trail of Bits’ Alessandro Gario provided a pre-release of the new version of osquery. For the past nine months Facebook had refactored osquery around Buck, a build system created and used by Facebook that had long been problematic for the greater community. Our pre-release restored CMake support, CI and packaging, and a few fixes not related to the build system.

Now the first full stable release of osquery is out! It’s a significant effort to improve the build system for the future of osquery, ensuring that:

Building osquery from source will no longer rely on Facebook to maintain and host the downloads for pre-built dependencies
The osquery project once again has a public-facing Continuous Integration server, automatically testing all proposed contributions to detect any developer mistakes or regressions
All contributors can use their preferred build tools: developers inside Facebook can use their build tool, Buck, and developers in the greater community can use the standard build tool, CMake
An all-new custom build toolchain for Linux will enable broader Linux support, and eventually reproducible builds

New features for osquery users:

The process_events table detects more kinds of events, on Linux
More powerful query syntax: osquery now supports a regex_match function to allow searches over a particular column of a given table
Initial support for eBPF-based event tracing, on Linux
New macOS tables for detecting T1/T2 chips, querying the list of running apps, and listing the installed Atom packages on macOS or Linux
The certificates, logged_in_users, and logical_drives tables on Windows are all greatly improved
Initial implementation of a new high-performance eventing framework that will enable more types of event-based monitoringImproved ability to profile and benchmark tables’ performance
New detections added to the macOS query pack

But wait, there’s more! Dozens of bugs have been squashed, additional security hardening mitigations have been turned on, certain performance cases have been improved and resource leaks plugged, the documentation has been updated…we could go on and on. For a full list of changes in this release, refer to the comprehensive change notes.

QueryCon and beyond

The hosts of the QueryCon 2019 posed for a team group shot!

We had so much fun hosting QueryCon this year and we want to thank everyone who attended. This event was a catalyst for positive change in our community thanks to the thoughts, discussions, and passion of this year’s attendees. We can’t wait to see how osquery improves now that its development has been unlocked.

What’s next for osquery? We want you to tell us! If you’re using osquery in your organization, let’s talk about what features and fixes should be next. Thanks to a revolutionary meeting of the minds, we now have the power to make it happen.