Please report any security issue or Envoy crash report to [email protected] where the issue will be triaged appropriately. Thank you in advance for helping to keep Envoy secure.
Envoy is a large growing community of volunteers, users, and vendors. The Envoy community has adopted this security disclosure and response policy to ensure we responsibly handle critical issues.
Security vulnerabilities should be handled quickly and sometimes privately. The primary goal of this process is to reduce the total time users are vulnerable to publicly known exploits.
The Product Security Team (PST) is responsible for organizing the entire response including internal communication and external disclosure but will need help from relevant developers to successfully run this process.
The initial Product Security Team will consist of all maintainers in the private envoy-security list. In the future we may decide to have a subset of maintainers work on security response given that this process is time consuming.
The Envoy community asks that all suspected vulnerabilities be privately and responsibly disclosed via the reporting policy.
If you know of a publicly disclosed security vulnerability please IMMEDIATELY email envoy-security to inform the Product Security Team (PST) about the vulnerability so they may start the patch, release, and communication process.
If possible the PST will ask the person making the public report if the issue can be handled via a private disclosure process (for example if the full exploit details have not yet been published). If the reporter denies the request for private disclosure, the PST will move swiftly with the fix and release process. In extreme cases GitHub can be asked to delete the issue but this generally isn't necessary and is unlikely to make a public disclosure less damaging.
For each vulnerability a member of the PST will volunteer to lead coordination with the "Fix Team" and is responsible for sending disclosure emails to the rest of the community. This lead will be referred to as the "Fix Lead."
The role of Fix Lead should rotate round-robin across the PST.
Note that given the current size of the Envoy community it is likely that the PST is the same as the "Fix team." (I.e., all maintainers). The PST may decide to bring in additional contributors for added expertise depending on the area of the code that contains the vulnerability.
All of the timelines below are suggestions and assume a private disclosure. The Fix Lead drives the schedule using their best judgment based on severity and development time. If the Fix Lead is dealing with a public disclosure all timelines become ASAP (assuming the vulnerability has a CVSS score >= 4; see below). If the fix relies on another upstream project's disclosure timeline, that will adjust the process as well. We will work with the upstream project to fit their timeline and best protect our users.
If the vulnerability affects the last point release version, e.g. 1.10, then the full security release process described in this document will be activated. A security point release will be created for 1.10, e.g. 1.10.1, together with a fix to main if necessary. Older point releases, e.g. 1.9, are not supported by the Envoy project and will not have any security release created.
If a security vulnerability affects only these older versions but not main or the last supported point release, the Envoy security team will share this information with the private distributor list, following the standard embargo process, but not create a security release. After the embargo expires, the vulnerability will be described as a GitHub issue. A CVE will be filed if warranted by severity.
If a vulnerability does not affect any point release but only main, additional caveats apply:
- If the issue is detected and a fix is available within 7 days of the introduction of the vulnerability, or the issue is deemed a low severity vulnerability by the Envoy maintainer and security teams, the fix will be publicly reviewed and landed on main. If the severity is at least medium or at maintainer discretion a courtesy e-mail will be sent to [email protected], [email protected], [email protected] and [email protected].
- If the vulnerability has been in existence for more than 7 days and is medium or higher, we will activate the security release process.
We advise distributors and operators working from the main branch to allow at least 5 days soak time after cutting a binary release before distribution or rollout, to allow time for our fuzzers to detect issues during their execution on ClusterFuzz. A soak period of 7 days provides an even stronger guarantee, since we will invoke the security release process for medium or higher severity issues for these older bugs.
See https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/security/threat_model. Vulnerabilities are evaluated against this threat model when deciding whether to activate the Envoy security release process.
These steps should be completed within the first 24 hours of disclosure.
- The Fix Lead will work quickly to identify relevant engineers from the affected projects and packages and CC those engineers into the disclosure thread. These selected developers are the Fix Team.
- The Fix Lead will get the Fix Team access to private security repos to develop the fix.
These steps should be completed within the 1-7 days of Disclosure.
- The Fix Lead and the Fix Team will create a CVSS using the CVSS Calculator. The Fix Lead makes the final call on the calculated CVSS; it is better to move quickly than making the CVSS perfect.
- The Fix Team will notify the Fix Lead that work on the fix branch is complete once there are LGTMs on all commits in the private repo from one or more maintainers.
If the CVSS score is under 4.0 (a low severity score) the Fix Team can decide to slow the release process down in the face of holidays, developer bandwidth, etc. These decisions must be discussed on the envoy-security mailing list.
A three week window will be provided to members of the private distributor list from candidate patch availability until the security release date. It is expected that distributors will normally be able to perform a release within this time window. If there are exceptional circumstances, the Envoy security team will raise this window to four weeks. The release window will be reduced if the security issue is public or embargo is broken.
We will endeavor not to overlap this three week window with or place it adjacent to major corporate holiday periods or end-of-quarter (e.g. impacting downstream Istio releases), where possible.
-
All reports to [email protected] will be triaged and have an initial response within 1 business day.
-
Privately disclosed issues will be fixed or publicly disclosed within 90 days by the Envoy security team. In exceptional circumstances we reserve the right to work with the discloser to coordinate on an extension, but this will be rarely used.
-
Any issue discovered by the Envoy security team and raised in our private bug tracker will be converted to a public issue within 90 days. We will regularly audit these issues to ensure that no major vulnerability (from the perspective of the threat model) is accidentally leaked.
-
Fuzz bugs are subject to a 90 day disclosure deadline.
-
Three weeks notice will be provided to private distributors from patch availability until the embargo deadline.
-
Public zero days will be fixed ASAP, but there is no SLO for this, since this will depend on the severity and impact to the organizations backing the Envoy security team.
With the fix development underway, the Fix Lead needs to come up with an overall communication plan for the wider community. This Disclosure process should begin after the Fix Team has developed a Fix or mitigation so that a realistic timeline can be communicated to users.
Disclosure of Forthcoming Fix to Users (Completed within 1-7 days of Disclosure)
- The Fix Lead will email [email protected] (CC [email protected]) informing users that a security vulnerability has been disclosed and that a fix will be made available at YYYY-MM-DD HH:MM UTC in the future via this list. This time is the Release Date.
- The Fix Lead will include any mitigating steps users can take until a fix is available.
The communication to users should be actionable. They should know when to block time to apply patches, understand exact mitigation steps, etc.
Optional Fix Disclosure to Private Distributors List (Completed within 1-14 days of Disclosure):
- The Fix Lead will make a determination with the help of the Fix Team if an issue is critical enough to require early disclosure to distributors. Generally this Private Distributor Disclosure process should be reserved for remotely exploitable or privilege escalation issues. Otherwise, this process can be skipped.
- The Fix Lead will email the patches to [email protected] so distributors can prepare builds to be available to users on the day of the issue's announcement. Any patches against main will be updated and resent weekly. Distributors should read about the Private Distributors List to find out the requirements for being added to this list.
- What if a vendor breaks embargo? The PST will assess the damage. The Fix Lead will make the call to release earlier or continue with the plan. When in doubt push forward and go public ASAP.
Fix Release Day (Completed within 1-21 days of Disclosure)
- The maintainers will create a new patch release branch from the latest patch release tag + the fix from the security branch. As a practical example if v1.5.3 is the latest patch release in Envoy.git a new branch will be created called v1.5.4 which includes only patches required to fix the issue.
- The Fix Lead will cherry-pick the patches onto the main branch and all relevant release branches. The Fix Team will LGTM and merge. Maintainers will merge these PRs as quickly as possible. Changes shouldn't be made to the commits even for a typo in the CHANGELOG as this will change the git sha of the commits leading to confusion and potentially conflicts as the fix is cherry-picked around branches.
- The Fix Lead will request a CVE from DWF and include the CVSS and release details.
- The Fix Lead will email envoy-{dev,users,announce}@googlegroups.com now that everything is public announcing the new releases, the CVE number, and the relevant merged PRs to get wide distribution and user action. As much as possible this email should be actionable and include links on how to apply the fix to user's environments; this can include links to external distributor documentation.
- The Fix Lead will remove the Fix Team from the private security repo.
The reporter of a vulnerability will be granted early access to fix patches upon request, prior to the general disclosure of patches to the Private Distributors List for the purpose of testing and internal vulnerability mitigation. They must accept the embargo policy below in order for this to occur. If the vulnerability reporter is also responsible for developing fix patches, they may make use of the patches internally in their organization at any point in the fix cycle.
These steps should be completed 1-3 days after the Release Date. The retrospective process should be blameless.
- The Fix Lead will send a retrospective of the process to [email protected] including details on everyone involved, the timeline of the process, links to relevant PRs that introduced the issue, if relevant, and any critiques of the response and release process.
- Maintainers and Fix Team are also encouraged to send their own feedback on the process to [email protected]. Honest critique is the only way we are going to get good at this as a community.
This list is intended to be used primarily to provide actionable information to multiple distribution vendors as well as a limited set of high impact end users at once. This list is not intended in the general case for end users to find out about security issues.
The information members receive on cncf-envoy-distributors-announce must not be made public, shared, nor even hinted at anywhere beyond the need-to-know within your specific team except with the list's explicit approval. This holds true until the public disclosure date/time that was agreed upon by the list. Members of the list and others may not use the information for anything other than getting the issue fixed for your respective users.
Before any information from the list is shared with respective members of your team required to fix said issue, they must agree to the same terms and only find out information on a need-to-know basis.
We typically expect a single point-of-contact (PoC) at any given legal entity. Within the organization, it is the responsibility of the PoC to share CVE and related patches internally. This should be performed on a strictly need-to-know basis with affected groups to the extent that this is technically plausible. All teams should be aware of the embargo conditions and accept them. Ultimately, if an organization breaks embargo transitively through such sharing, they will lose the early disclosure privilege, so it's in their best interest to carefully share information internally, following best practices and use their judgement in balancing the tradeoff between protecting users and maintaining confidentiality.
The embargo applies to information shared, source code and binary images. It is a violation of the embargo policy to share binary distributions of the security fixes before the public release date. This includes, but is not limited to, Envoy binaries and Docker images. It is expected that distributors have a method to stage and validate new binaries without exposing them publicly.
If the information shared is under embargo from a third party, where Envoy is one of many projects that a disclosure is shared with, it is critical to consider that the ramifications of any leak will extend beyond the Envoy community and will leave us in a position in which we will be less likely to receive embargoed reports in the future.
In the unfortunate event you share the information beyond what is allowed by this policy, you must urgently inform the [email protected] mailing list of exactly what information leaked and to whom. A retrospective will take place after the leak so we can assess how to prevent making the same mistake in the future.
If you continue to leak information and break the policy outlined here, you will be removed from the list.
This is a team effort. As a member of the list you must carry some water. This could be in the form of the following:
Technical
- Review and/or test the proposed patches and point out potential issues with them (such as incomplete fixes for the originally reported issues, additional issues you might notice, and newly introduced bugs), and inform the list of the work done even if no issues were encountered.
Administrative
- Help draft emails to the public disclosure mailing list.
- Help with release notes.
To be eligible for the cncf-envoy-distributors-announce mailing list, your use of Envoy should:
-
Be either:
- An actively maintained distribution of Envoy components. An example is "SuperAwesomeLinuxDistro" which offers Envoy pre-built packages. Another example is "SuperAwesomeServiceMesh" which offers a service mesh product that includes Envoy as a component.
OR
- Offer Envoy as a publicly available infrastructure or platform service, in which the product clearly states (e.g. public documentation, blog posts, marketing copy, etc.) that it is built on top of Envoy. E.g., "SuperAwesomeCloudProvider's Envoy as a Service (EaaS)". An infrastructure service that uses Envoy for a product but does not publicly say they are using Envoy does not generally qualify (see option 3 that follows). This is essentially IaaS or PaaS. If you use Envoy to support a SaaS, e.g. "SuperAwesomeCatVideoService", this does not generally qualify.
OR
- An end user of Envoy that satisfies the following requirements:
- Is "well known" to the Envoy community. Being "well known" is fully subjective and determined by the Envoy maintainers and security team. Becoming "well known" would generally be achieved by activities such as: PR contributions, either code or documentation; helping other end users on Slack, GitHub, and the mailing lists; speaking about use of Envoy at conferences; writing about use of Envoy in blog posts; sponsoring Envoy conferences, meetups, and other activities; etc. This is a more strict variant of item 5 below.
- Is of sufficient size, scale, and impact to make your inclusion on the list worthwhile. The definition of size, scale, and impact is fully subjective and determined by the Envoy maintainers and security team. The definition will not be discussed further in this document.
- You must smoke test and then widely deploy security patches promptly and report back success or failure ASAP. Furthermore, the Envoy maintainers may occasionally ask you to smoke test especially risky public PRs before they are merged. Not performing these tasks in a reasonably prompt timeframe will result in removal from the list. This is a more strict variant of item 7 below.
- In order to balance inclusion in the list versus a greater chance of accidental disclosure, end users added to the list via this option will be limited to a total of 10 slots. Periodic review (see below) may allow new slots to open, so please continue to apply if it seems your organization would otherwise qualify. The security team also reserves the right to change this limit in the future.
-
Have a user or customer base not limited to your own organization (except for option 3 above). We will use the size of the user or customer base as part of the criteria to determine eligibility.
-
Have a publicly verifiable track record up to present day of fixing security issues.
-
Not be a downstream or rebuild of another distribution.
-
Be a participant and active contributor in the community.
-
Accept the Embargo Policy that is outlined above. You must have a way to privately stage and validate your updates that does not violate the embargo.
-
Be willing to contribute back as outlined above.
-
Be able to perform a security release of your product within a three week window from candidate fix patch availability.
-
Have someone already on the list vouch for the person requesting membership on behalf of your distribution.
-
Nominate an e-mail alias or list for your organization to receive updates. This should not be an individual user address, but instead a list that can be maintained by your organization as individuals come and go. A good example is [email protected], a bad example is [email protected]. You must accept the invite sent to this address or you will not receive any e-mail updates. This e-mail address will be shared with the Envoy community.
Note that Envoy maintainers are members of the Envoy security team. Members of the Envoy security team and the organizations that they represent are implicitly included in the private distributor list. These organizations do not need to meet the above list of criteria with the exception of the acceptance of the embargo policy.
New membership requests are sent to [email protected].
In the body of your request please specify how you qualify and fulfill each criterion listed in Membership Criteria.
Here is a pseudo example:
To: [email protected]
Subject: Seven-Corp Membership to cncf-envoy-distributors-announce
Below are each criterion and why I think we, Seven-Corp, qualify.
> 1. Be an actively maintained distribution of Envoy components OR offer Envoy as a publicly
available service in which the product clearly states that it is built on top of Envoy OR
be a well known end user of sufficient size, scale, and impact to make your
inclusion worthwhile.
We distribute the "Seven" distribution of Envoy [link]. We have been doing
this since 1999 before proxies were even cool.
OR
We use Envoy for our #1 rated cat video service and have 40 billion MAU, proxying 40 trillion^2 RPS
through Envoy at the edge. Secure cat videos are our top priority. We also contribute a lot to the Envoy
community by implementing features, not making Matt ask for documentation or tests, and writing blog
posts about efficient Envoy cat video serving.
> 2. Have a user or customer base not limited to your own organization. Please specify an
> approximate size of your user or customer base, including the number of
> production deployments.
Our user base spans of the extensive "Seven" community. We have a slack and
GitHub repos and mailing lists where the community hangs out. We have ~2000
customers, of which approximately 400 are using Seven in production. [links]
> 3. Have a publicly verifiable track record up to present day of fixing security
issues.
We announce on our blog all upstream patches we apply to "Seven." [link to blog
posts]
> 4. Not be a downstream or rebuild of another distribution. If you offer Envoy as a publicly
> available infrastructure or platform service, this condition does not need to apply.
This does not apply, "Seven" is a unique snowflake distribution.
> 5. Be a participant and active contributor in the community.
Our members, Acidburn, Cereal, and ZeroCool are outstanding members and are well
known throughout the Envoy community. Especially for their contributions
in hacking the Gibson.
> 6. Accept the Embargo Policy that is outlined above. You must
have a way to privately stage and validate your updates that does not violate
the embargo.
We accept.
> 7. Be willing to contribute back as outlined above.
We are definitely willing to help!
> 8. Be able to perform a security release of your product within a three week window from candidate fix
patch availability.
We affirm we can spin out new security releases within a 2 week window.
> 9. Have someone already on the list vouch for the person requesting membership
> on behalf of your distribution.
CrashOverride will vouch for the "Seven" distribution joining the distribution list.
> 10. Nominate an e-mail alias or list for your organization to receive updates. This should not be
an individual user address, but instead a list that can be maintained by your organization as
individuals come and go. A good example is [email protected], a bad example is
[email protected]. You must accept the invite sent to this address or you will not receive any
e-mail updates. This e-mail address will be shared with the Envoy community.
[email protected]
In all cases, members of the distribution list will be reviewed on a yearly basis by the maintainers and security team to ensure they still qualify for inclusion on the list.
| Organization | End User | Last Review |
|---|---|---|
| Aspen Mesh | No | 12/19 |
| AWS | No | 12/19 |
| Cilium | No | 12/19 |
| Cloud Foundry | No | 12/19 |
| Datawire | No | 12/19 |
| No | 12/19 | |
| IBM | No | 12/19 |
| Istio | No | 12/19 |
| Microsoft | No | 2/21 |
| Red Hat | No | 12/19 |
| solo.io | No | 12/19 |
| Tetrate | No | 12/19 |
| VMware | No | 12/19 |
| Yes | 12/19 | |
| Dropbox | Yes | 01/20 |
| Stripe | Yes | 01/20 |
| Square | Yes | 05/21 |
| Apple | Yes | 05/21 |
| Spotify | Yes | 06/21 |