diff --git a/sql/2021/javascript/web_components_pct.sql b/sql/2021/javascript/web_components_pct.sql new file mode 100644 index 00000000000..a6e3b48d184 --- /dev/null +++ b/sql/2021/javascript/web_components_pct.sql @@ -0,0 +1,24 @@ +SELECT + client, + custom_elements, + shadow_roots, + templates, + total, + custom_elements / total AS pct_custom_elements, + shadow_roots / total AS pct_shadow_roots, + templates / total AS pct_templates +FROM ( + SELECT + client, + COUNT(0) AS total, + COUNTIF(ARRAY_LENGTH(JSON_EXTRACT_ARRAY(js, '$.web_component_specs.custom_elements')) > 0) AS custom_elements, + COUNTIF(ARRAY_LENGTH(JSON_EXTRACT_ARRAY(js, '$.web_component_specs.shadow_roots')) > 0) AS shadow_roots, + COUNTIF(ARRAY_LENGTH(JSON_EXTRACT_ARRAY(js, '$.web_component_specs.template')) > 0) AS templates + FROM ( + SELECT + _TABLE_SUFFIX AS client, + JSON_EXTRACT_SCALAR(payload, '$._javascript') AS js + FROM + `httparchive.pages.2021_09_01_*`) + GROUP BY + client) diff --git a/src/config/2021.json b/src/config/2021.json index c3345f2cc96..2ed5d3aa9ab 100644 --- a/src/config/2021.json +++ b/src/config/2021.json @@ -22,8 +22,7 @@ "part": "I", "chapter_number": "2", "title": "JavaScript", - "slug": "javascript", - "todo": "true" + "slug": "javascript" }, { "part": "I", diff --git a/src/content/en/2021/javascript.md b/src/content/en/2021/javascript.md index 762ed8a3ec3..d0c5cb88990 100644 --- a/src/content/en/2021/javascript.md +++ b/src/content/en/2021/javascript.md @@ -5,7 +5,7 @@ description: JavaScript chapter of the 2021 Web Almanac covering the usage of Ja authors: [NishuGoel] reviewers: [soulcorrosion, mgechev, rviscomi, pankajparkar, tunetheweb] analysts: [pankajparkar, max-ostapenko, rviscomi] -editors: [rviscomi, pankajparkar] +editors: [rviscomi, pankajparkar, shantsis] translators: [] results: https://docs.google.com/spreadsheets/d/1zU9rHpI3nC6jTz3xgN6w13afW7x34xAKBh2IPH-lVxk/ NishuGoel_bio: Nishu Goel is an engineer at Web DataWorks. She is a Google Developer Expert for Web Technologies and Angular, Microsoft MVP for Developer Technologies, and the author of Step by Step Guide Angular Routing (BPB, 2019) and A Hands-on Guide to Angular (Educative, 2021). Find her writings at unravelweb.dev. @@ -16,22 +16,21 @@ featured_stat_2: 45.6% featured_stat_label_2: Images are the most requested content type on pages loaded by JavaScript. featured_stat_3: 2.7% featured_stat_label_3: Percent of desktop pages using custom elements. -unedited: true --- ## Introduction -The speed and consistency that the JavaScript language has evolved with over the past years is tremendous. While in the past it was used primarily for client-side, it has taken a very important and respected place in the world of building services and server-side tools. JavaScript has evolved to a point where it is not only possible to create faster applications but also to run servers within browsers. +The speed and consistency at which the JavaScript language has evolved over the past years is tremendous. While in the past it was used primarily on the client side, it has taken a very important and respected place in the world of building services and server-side tools. JavaScript has evolved to a point where it is not only possible to create faster applications but also to run servers within browsers. -There is a lot that happens in the browser when rendering the application, from downloading JavaScript to parsing, compiling and executing it. Let's start with that first step and try to understand how much JavaScript is actually requested by pages. +There is a lot that happens in the browser when rendering the application, from downloading JavaScript to parsing, compiling, and executing it. Let's start with that first step and try to understand how much JavaScript is actually requested by pages. ## How much JavaScript do we load? -To measure is the key towards improvement, they say. To improve the usage of JavaScript in our applications, we need to measure how much JavaScript being shipped is actually required. Let's dig in to understand the distribution of JavaScript bytes per page, considering what a major role it plays in the web setup. +They say, "to measure is the key towards improvement". To improve the usage of JavaScript in our applications, we need to measure how much of the JavaScript being shipped is actually required. Let's dig in to understand the distribution of JavaScript bytes per page, considering what a major role it plays in the web setup. {{ figure_markup( image="javascript-bytes-per-page.png", - caption="Distribution of the amount of JavaScript kilobytes loaded per page.", + caption="Distribution of the amount of JavaScript loaded per page.", description="Bar chart showing the distribution of JavaScript bytes per page. The median desktop page loads 463 kilobytes of JavaScript. The 10th, 25th, 50th, 75th, and 90th percentiles for desktop are: 94 KB, 220 KB, 463 KB, 852 KB, and 1,354 KB.", chart_url="https://docs.google.com/spreadsheets/d/e/2PACX-1vTpHzC_cMZYj2VLzQ4ODK3uvZkNBXtwdAZriZaBwjLjUM1SGwwmJs9rv8T6OtNdXox29PQ34CasUUwc/pubchart?oid=329775434&format=interactive", sheets_gid="18398250", @@ -39,36 +38,34 @@ To measure is the key towards improvement, they say. To improve the usage of Jav ) }} -The 50th percentile (median) desktop page loads 463 kilobytes of JavaScript, whereas a median page load on a mobile device sends 427 kilobytes. +The 50th percentile (median) mobile page loads 427 KB of JavaScript, whereas the median page loaded on a desktop device sends 463 KB. -Compared to [2019's results](../2019/javascript#fig-2), this shows an increase of 18.4% in the usage of JavaScript for desktop devices and an increase of 18.9% on mobile devices. As we see the trend is towards using more JavaScript over the years, this could slow down the rendering of an application given the additional CPU work. It's worth to note that these statistics represent the transferred bytes which could be compressed responses and thus, the actual cost to the CPU could be significantly higher. +Compared to [2019's results](../2019/javascript#fig-2), this shows an increase of 18.4% in the usage of JavaScript for desktop devices and an increase of 18.9% on mobile devices. The trend over time is moving towards using more JavaScript, which could slow down the rendering of an application given the additional CPU work. It's worth noting that these statistics represent the transferred bytes which could be compressed responses and thus, the actual cost to the CPU could be significantly higher. Let's have a look at how much JavaScript is actually required to be loaded on the page. {{ figure_markup( image="unused-javascript-bytes-per-page.png", caption="Distribution of the amount of unused JavaScript bytes on mobile.", - description="Bar chart showing the distribution of unused JavaScript bytes per page. The median mobile loads 155 KBs of JavaScript that is unused. The 10th, 25th, 50th, 75th, and 90th percentiles for mobile are: 20KB, 64KB, 155KB, 329KB, 598KB.", + description="Bar chart showing the distribution of unused JavaScript bytes per page. The median mobile loads 155 KB of JavaScript that is unused. The 10th, 25th, 50th, 75th, and 90th percentiles for mobile are: 20 KB, 64 KB, 155 KB, 329 KB, 598 KB.", chart_url="https://docs.google.com/spreadsheets/d/e/2PACX-1vTpHzC_cMZYj2VLzQ4ODK3uvZkNBXtwdAZriZaBwjLjUM1SGwwmJs9rv8T6OtNdXox29PQ34CasUUwc/pubchart?oid=656542402&format=interactive", sheets_gid="1704839581", sql_file="unused_js_bytes_distribution.sql" ) }} -The median page loads 155 kilobytes of unused JavaScript and at the 90th percentile, 598 kilobytes of JavaScript are unused. +According to [Lighthouse](./methodology#lighthouse), the median mobile page loads 155 KB of unused JavaScript. And at the 90th percentile, 598 KB of JavaScript are unused. {{ figure_markup( image="unused-vs-total-javascript.png", - caption="Percent of JavaScript loaded vs. JavaScript unused on mobile page.", - description="Bar chart showing the difference in the loaded JavaScript and the unused JavaScript from the total loaded. Out of 427KB of loaded JavaScript on a median mobile page, 155KB is unused. 36.2% of the total loaded JavaScript goes unused adding to the CPU cost.", + caption="Distribution of unused and total JavaScript bytes on mobile pages.", + description="Bar chart showing the difference in the loaded JavaScript and the unused JavaScript from the total loaded. Out of 427 KB of loaded JavaScript on a median mobile page, 155 KB is unused. 36.2% of the total loaded JavaScript goes unused adding to the CPU cost.", chart_url="https://docs.google.com/spreadsheets/d/e/2PACX-1vTpHzC_cMZYj2VLzQ4ODK3uvZkNBXtwdAZriZaBwjLjUM1SGwwmJs9rv8T6OtNdXox29PQ34CasUUwc/pubchart?oid=890651092&format=interactive", sheets_gid="1521645399", sql_file="unused_js_bytes_distribution.sql" ) }} -This contributes to 36.2% of unused JavaScript on a page. - {{ figure_markup( content="36.2%", caption="Percent unused from the total loaded JavaScript", @@ -78,17 +75,17 @@ This contributes to 36.2% of unused JavaScript on a page. ) }} -This is such a significant figure to be using the CPU with other important resources and to just go to waste, given the impact it can have on the Largest Contentful Paint (LCP) of the page, especially for the mobile users with limited data plans. This could be a result of a lot of boilerplate code that is shipped when using large frameworks or libraries. +To put it another way, 36.2% of JavaScript bytes on the median mobile page go unused. Given the impact JavaScript can have on the Largest Contentful Paint (LCP) of the page, especially for mobile users with limited device capabilities and data plans, this is such a significant figure to be consuming CPU cycles with other important resources just to go to waste. Such wastefulness could be the result of a lot of unused boilerplate code that gets shipped with large frameworks or libraries. -One step towards improving the percentage is to look at the percentage of unused JavaScript when checking the lighthouse report for the page and finding opportunities to get rid of the unnecessary JavaScript. +Site owners could reduce the percentage of wasted JavaScript bytes by using Lighthouse to check for unused JavaScript and follow best practices to remove unused code. ### JavaScript requests per page -One of the contributing factors towards slow rendering of the web page could be the requests made on the page, especially when they are blocking. It would be interesting to look at the number of JavaScript requests made per page on both desktop and mobile devices. +One of the contributing factors towards slow rendering of the web page could be the requests made on the page, especially when they are blocking requests. It's therefore of interest to look at the number of JavaScript requests made per page on both desktop and mobile devices. {{ figure_markup( image="js-requests-per-page.png", - caption="Distribution of JavaScript requests on desktop and mobile pages.", + caption="Distribution of the number of JavaScript requests per page.", description="Bar chart showing the distribution of JavaScript requests on desktop and mobile pages. The median mobile page loads 20 JavaScript resources as compared to 21 JavaScript resources on desktop. The 10th, 25th, 50th, 75th, and 90th percentiles for requests made on mobile are: 4, 10, 20, 35, 56", chart_url="https://docs.google.com/spreadsheets/d/e/2PACX-1vTpHzC_cMZYj2VLzQ4ODK3uvZkNBXtwdAZriZaBwjLjUM1SGwwmJs9rv8T6OtNdXox29PQ34CasUUwc/pubchart?oid=1487796178&format=interactive", sheets_gid="159538568", @@ -98,21 +95,21 @@ One of the contributing factors towards slow rendering of the web page could be The median desktop page loads 21 JavaScript resources (`.js` and `.mjs` requests), going up to 59 resources at the 90th percentile. -As compared to the [last year's results](../2020/javascript#request-count), there has been a marginal increase in the number of JavaScript resources requested in 2021, with the median number of JavaScript resources loaded being 20 for desktop pages and 19 for mobile. - {{ figure_markup( image="js-resources-over-years.png", - caption="Distribution of JavaScript resources loaded over desktop and mobile devices by year.", - description="Bar chart showing the distribution of JavaScript resources loaded over desktop and mobile devices by year. In 2020, the median JS requests made on a page were 39 and this has increased to 41 in 2021.", + caption="Distribution of the number of JavaScript requests per page by year.", + description="Bar chart showing the distribution of JavaScript resources loaded over desktop and mobile devices by year. In 2020, the median JS requests made on a page were 19 and this has increased to 20 in 2021.", chart_url="https://docs.google.com/spreadsheets/d/e/2PACX-1vTpHzC_cMZYj2VLzQ4ODK3uvZkNBXtwdAZriZaBwjLjUM1SGwwmJs9rv8T6OtNdXox29PQ34CasUUwc/pubchart?oid=882068136&format=interactive", sheets_gid="1068898615", sql_file="requests_2020.sql" ) }} +As compared with [last year's results](../2020/javascript#request-count), there has been a marginal increase in the number of JavaScript resources requested in 2021, with the median number of JavaScript resources loaded being 20 for desktop pages and 19 for mobile. + The trend is gradually increasing in the number of JavaScript resources loaded on a page. This would make one wonder if the number should actually increase or decrease considering that fewer JavaScript requests might lead to better performance in some cases but not in others. -This is where the recent advances in the HTTP protocol come in and the idea of reducing the number of JavaScript requests for better performance gets inaccurate. With the introduction of HTTP/2 and HTTP/3 the overhead of HTTP requests has significantly reduced so requesting the same resources over more requests is not necessarily a bad thing anymore. Read more about these protocols in the [HTTP](./http) chapter. +This is where the recent advances in the HTTP protocol come in and the idea of reducing the number of JavaScript requests for better performance gets inaccurate. With the introduction of HTTP/2 and HTTP/3, the overhead of HTTP requests has been significantly reduced, so requesting the same resources over more requests is not necessarily a bad thing anymore. To learn more about these protocols, see the [HTTP chapter](./http). ## How is JavaScript requested? @@ -120,9 +117,9 @@ JavaScript can be loaded into a page in a number of different ways, and how it i ### `module` and `nomodule` -When loading a website, the browser renders the HTML and requests the appropriate resources. It consumes the polyfills referenced in the code for the effective rendering and functioning of the page. The modern browsers that support newer syntax like [arrow functions](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/Arrow_functions), [async functions](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function), etc. do not need loads of polyfills to make things work and therefore, should not have to. +When loading a website, the browser renders the HTML and requests the appropriate resources. It consumes the polyfills referenced in the code for the effective rendering and functioning of the page. Modern browsers that support newer syntax like [arrow functions](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/Arrow_functions) and [async functions](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function) do not need loads of polyfills to make things work and therefore, should not have to. -This is when differential loading takes care of things. Specifying the `type="module"` attribute would serve the modern browsers the bundle with modern syntax and fewer polyfills, if any. Similarly, older browsers that lack support for modules will be served the bundle with required polyfills and transpiled code syntax with attribute `type="nomodule"`. Read more about [the usage of module/nomodule here](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules#applying_the_module_to_your_html) +This is when differential loading takes care of things. Specifying the `type="module"` attribute would serve the modern browsers the bundle with modern syntax and fewer polyfills, if any. Similarly, older browsers that lack support for modules will be served the bundle with required polyfills and transpiled code syntax with the `type="nomodule"` attribute. Read more about [the usage of module/nomodule](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules#applying_the_module_to_your_html). Let's look at the data to understand the adoption of these attributes. @@ -130,81 +127,93 @@ Let's look at the data to understand the adoption of these attributes. - - - + + + - + - + - - + +
Client`module``nomodule`AttributeDesktopMobile
Desktop`module` 4.6%3.9%4.3%
Mobile4.3%`nomodule`3.9% 3.9%
-
{{ figure_link(caption="Distribution of differential loading usage on desktop and mobile clients.", sheets_gid="1261294750", sql_file="module_and_nomodule.sql") }}
+
{{ figure_link( + caption="Distribution of differential loading usage on desktop and mobile clients.", + sheets_gid="1261294750", + sql_file="module_and_nomodule.sql" + ) }}
-4.6% of desktop pages use the attribute `type="module"` whereas only 3.9% of mobile pages use `type="nomodule"`. This could be due to the fact that the mobile dataset being much larger contains more "long-tail" websites that might not be using the latest features. +4.6% of desktop pages use the `type="module"` attribute, whereas only 3.9% of mobile pages use `type="nomodule"`. This could be due to the fact that the mobile dataset being [much larger](./methodology#websites) contains more "long-tail" websites that might not be using the latest features. -It is important to note that with the end of support for IE11 browser, differential loading is less applicable because evergreen browsers support modern JavaScript syntax. One of the JavaScript frameworks, Angular, for example, removed support for the legacy browsers with Angular v13 released November 2021. +It is important to note that with the end of support for IE 11 browser, differential loading is less applicable because evergreen browsers support modern JavaScript syntax. The Angular framework, for example, removed support for legacy browsers in Angular v13, which was released November 2021. ### `async` and `defer` -JavaScript loading could be render blocking unless it is specified as deferred or async. This is one of the contributing factors to the slow website loading as often JavaScript (or at least not all the JavaScript) is needed for the initial render. +JavaScript loading could be render-blocking unless it is specified as asynchronous or deferred. This is one of the contributing factors to slow performance, as oftentimes JavaScript (or at least some of it) is needed for the initial render. -However, loading the JavaScript asynchronously or deferring it helps in some ways to improve this experience. The `async` and `defer` attribute load the scripts asynchronously. The scripts with `async` attribute are executed irrespective of the order in which they are defined, however, `defer` executes the scripts only after the document is completely parsed and ensures their execution will take place in the specified order. We will see the statistics for how many pages actually specify these attributes for the JavaScript requested in the browser. +However, loading the JavaScript asynchronously or deferred helps in some ways to improve this experience. Both the `async` and `defer` attributes load the scripts asynchronously. The scripts with the `async` attribute are executed irrespective of the order in which they are defined, however, `defer` executes the scripts only after the document is completely parsed, ensuring that their execution will take place in the specified order. Let's look at how many pages actually specify these attributes for the JavaScript requested in the browser.
- - - - + + + - + - - + - - + + + + + + + + + + +
Client`async``defer`NeitherAttributeDesktopMobile
Desktop`async` 89.3%48.1%10.3%89.1%
Mobile89.1%`defer`48.1% 47.8%
Both35.7%35.6%
Neither10.3% 10.4%
-
{{ figure_link(caption="Percent of pages using `async`, `defer`, or neither.", sheets_gid="2038121216", sql_file="breakdown_of_pages_using_async_defer.sql") }}
+
{{ figure_link( + caption="Percent of pages using `async` and `defer`.", + sheets_gid="2038121216", + sql_file="breakdown_of_pages_using_async_defer.sql" + ) }}
-There was an anti-pattern observed in the last year's results that some websites use both `async` and `defer` attribute resulting in fallback to `async` if the browser supports it and using `defer` for IE8 and IE9 browsers. This is, however, unnecessary now for most of the sites since `async` takes the precedence on all suppoting browsers and this pattern in turn, interrupts the HTML parsing, instead of deferring until the page has loaded. The usage was so frequent that [11.4%](../2020/javascript#how-do-we-load-our-javascript) of mobile pages were seen with at least one script with `async` and `defer` attributes used together. The [root causes](https://twitter.com/rick_viscomi/status/1331735748060524551?s=20) were found and an action item was also taken down to [remove such usage going forward](https://twitter.com/Kraft/status/1336772912414601224?s=20). +There was an anti-pattern observed in last year's results that some websites use both `async` and `defer` attributes on the same script, which falls back to `async` if the browser supports it and using `defer` for IE 8 and IE 9 browsers. This is, however, unnecessary now for most of the sites since `async` takes precedence on all supported browsers and. In turn, this pattern interrupts HTML parsing instead of deferring until the page has loaded. The usage was so frequent that [11.4%](../2020/javascript#how-do-we-load-our-javascript) of mobile pages were seen with at least one script with `async` and `defer` attributes used together. The [root causes](https://twitter.com/rick_viscomi/status/1331735748060524551?s=20) were found and an action item was also taken down to [remove such usage going forward](https://twitter.com/Kraft/status/1336772912414601224?s=20). -This year, we improved our measures of calculating these figures by looking at the rendered DOM, as opposed to the initial HTML. {{ figure_markup( content="35.6%", - caption="Percent of pages using the `async` and `defer` attribute together.", + caption="Percent of mobile pages on which the `async` and `defer` attributes are set on the same script.", classes="big-number", sheets_gid="2038121216", sql_file="breakdown_of_pages_using_async_defer.sql" ) }} -It is found that 35.6% of mobile pages use the `async` and `defer` attributes together. This was counted using a custom metric that measures the usage of these attributes on the rendered DOM. Based on the custom metric, the variation is a lot as compared to the previous year. When we dived deeper to understand this gap, it was found that a lot of pages update these attributes dynamically after the pages have already been loaded. This is one reason why more results are found using the results from the custom metric (rendered DOM as opposed to initial DOM). - -A typical example website, loads the below script: +This year, we found that 35.6% of mobile pages use the `async` and `defer` attributes together. The large discrepancy from last year is due to a methodological improvement to measure attribute usage at runtime, rather than parsing the static contents of the initial HTML. This difference shows that many pages update these attributes dynamically after the document has already been loaded. For example, one website was found to include the following script: ```js @@ -217,25 +226,29 @@ A typical example websit ``` -According to [Wikipedia](https://en.wikipedia.org/wiki/Matomo_(software)), Piwik (now known as Matomo): -
is a free and open source web analytics application developed by a team of international developers, that runs on a PHP/MySQL webserver. It tracks online visits to one or more websites and displays reports on these visits for analysis. As of June 2018, Matomo was used by over 1,455,000 websites, or 1.3% of all websites with known traffic analysis tools...
+So, what is Piwik? According to its Wikipedia entry: -This information strongly suggests that a lot of the increase is due to these marketing and analytics providers that dynamically inject these async/defer scripts into the page. +
+
Matomo, formerly Piwik, is a free and open source web analytics application developed by a team of international developers, that runs on a PHP/MySQL web server. It tracks online visits to one or more websites and displays reports on these visits for analysis. As of June 2018, Matomo was used by over 1,455,000 websites, or 1.3% of all websites with known traffic analysis tools…
+
— Matomo (software) on Wikipedia
+
+ +This information strongly suggests that much of the increase we observed may be due to similar marketing and analytics providers that dynamically inject these `async` and `defer` scripts into the page later than had been previously detected. {{ figure_markup( content="2.6%", - caption="Percent of requests using the `async` and `defer` attribute together.", + caption="Percent of scripts using the `async` and `defer` attribute together.", classes="big-number", sheets_gid="655357075", sql_file="breakdown_of_scripts_using_async_defer_module_nomodule.sql" ) }} -Even though a large percentage of pages use this anti-pattern, it turns out only 2.6% of requests use both `async` and `defer` on the same script element. +Even though a large percentage of pages use this anti-pattern, it turns out that only 2.6% of all scripts use both `async` and `defer` on the same script element. ### First-party vs third-party -The median number of JavaScript resources requested on mobile pages, as [observed earlier](#how-much-javascript-do-we-load), is 20. Let us now look into which ones of these are first-party or third-party requests. +Recall from the [How much JavaScript do we load](#how-much-javascript-do-we-load) section that the median number of JavaScript requests on mobile pages is 20. In this section, we'll take a look at the breakdown of first and third-party JavaScript requests. {{ figure_markup( image="js-requests-mobile-host.png", @@ -247,7 +260,7 @@ The median number of JavaScript resources requested on mobile pages, as [observe ) }} -A median of mobile pages request 10 third-party resources whereas 9 first-party requests. let's look at how the results vary for the desktop client. +The median mobile page requests 10 third-party resources and 9 first-party requests. This difference increases as we move up to the 90th percentile as 33 requests on mobile pages are first-party but the number goes up to 34 for third-party requests for the mobile pages. Clearly, the number of third-party resources requested is always one step ahead of the first-party ones. {{ figure_markup( image="js-requests-desktop-host.png", @@ -259,16 +272,13 @@ A median of mobile pages request 10 third-party resources whereas 9 first-party ) }} -A median of desktop pages request 11 third-party resources whereas 10 first-party requests. This difference increases as we move up to the 90th percentile as 33 requests on mobile pages are first party but the number goes up to 34 for third-party requests for the mobile pages. Clearly, the number of third-party resources requested is always one step ahead of the first-party ones. Irrespective of the performance and reliability risks that requesting third-party resources brings, the usage seems to favor third-party scripts, which could be due to the useful interactivity features that it gives to the web. +The median desktop page requests 11 third-party resources, compared to 10 first-party requests. Irrespective of the performance and reliability risks that third-party resources may bring, both desktop and mobile pages consistently seem to favor third-party scripts. This effect could be due to the useful interactivity features that third-party scripts give to the web. -This is where one has to put their 'performance-nerd' cap on, and ensure that using third-party scripts doesn't result in losing control over the performance of the page or the main thread getting bloated with too much JavaScript being transferred by loading these scripts better. +Nevertheless, site owners must ensure that their third-party scripts are loaded performantly. Harry Roberts advocates for going a step further and stress testing third-parties for performance and resilience. -### Preload and Prefetch +### `preload` and `prefetch` -At the render of a page, the browser downloads the given resources and to prioritise the download of some resources over others the browser uses [resource hints](../resource-hints). The resource hints, preload and prefetch, are basically to tell the browser to either download some resource right away or to download it whenever it can to be able to present the resource when required. - -The `preload` hint tells the browser to download the resource with a higher priority as it will be required on the current page. -The `prefetch` hint, however, tells the browser that the resource could be required after some time (useful for future navigation) and it'd better to fetch it when the browser has the capacity to do so and make it available as soon as it is required. +As a page is rendered, the browser downloads the given resources and prioritizes the download of some resources the browser uses over others using resource hints. The `preload` hint tells the browser to download the resource with a higher priority as it will be required on the current page. The `prefetch` hint, however, tells the browser that the resource could be required after some time (useful for future navigation) and it'd better to fetch it when the browser has the capacity to do so and make it available as soon as it is required. Learn more about how these features are used in the [Resource Hints](./resource-hints) chapter. {{ figure_markup( image="javascript-resource-hint-usage.png", @@ -280,111 +290,90 @@ The `prefetch` hint, however, tells the browser that the resource could be requi ) }} -The preload hints are used by 15.4% of mobile pages to load JavaScript, whereas only 1.0% of mobile pages use the prefetch hint. Overall, 15.8% and 1.1% repectively of desktop pages use these resource hints to load the JavaScript resources. +`preload` hints are used to load JavaScript on 15.4% of mobile pages, whereas only 1.0% of mobile pages use the `prefetch` hint. 15.8% and 1.1% of desktop pages use these resource hints to load JavaScript resources, respectively. -It would also be ideal to see how many preload and prefetch hints are used per page as that affects the impact of these hints. For example, there are 5 resources to be loaded on the render and all five use the preload hint, the preload hint would try to prioritize all the resources and this would then work as it would have without the preload hint. +It would also be useful to see how many `preload` and `prefetch` hints are used per page, as that affects the impact of these hints. For example, if there are five resources to be loaded on the render and all five use the `preload` hint, the browser would try to prioritize every resource, which would effectively work as if no `preload` hint was used at all. {{ figure_markup( - image="prefetch-hints-per-page.png", - caption="Distribution of prefetch hints per page.", - description="Bar chart showing the distribution of prefetch hints per page. A median desktop page loads 2 JavaScript resources with the prefetch hint.", - chart_url="https://docs.google.com/spreadsheets/d/e/2PACX-1vTpHzC_cMZYj2VLzQ4ODK3uvZkNBXtwdAZriZaBwjLjUM1SGwwmJs9rv8T6OtNdXox29PQ34CasUUwc/pubchart?oid=1265752379&format=interactive", + image="preload-hints-per-page.png", + caption="Distribution of preload hints for JavaScript resources per page.", + description="Bar chart showing the distribution of prefetch hints per page. A median desktop page loads 1 JavaScript resource with the preload hint.", + chart_url="https://docs.google.com/spreadsheets/d/e/2PACX-1vTpHzC_cMZYj2VLzQ4ODK3uvZkNBXtwdAZriZaBwjLjUM1SGwwmJs9rv8T6OtNdXox29PQ34CasUUwc/pubchart?oid=1180601651&format=interactive", sheets_gid="1107832831", sql_file="resource-hints-prefetch-preload-percentage.sql" ) }} {{ figure_markup( - image="preload-hints-per-page.png", - caption="Distribution of preload hints per page.", - description="Bar chart showing the distribution of prefetch hints per page. A median desktop page loads 1 JavaScript resource with the preload hint.", - chart_url="https://docs.google.com/spreadsheets/d/e/2PACX-1vTpHzC_cMZYj2VLzQ4ODK3uvZkNBXtwdAZriZaBwjLjUM1SGwwmJs9rv8T6OtNdXox29PQ34CasUUwc/pubchart?oid=1180601651&format=interactive", + image="prefetch-hints-per-page.png", + caption="Distribution of prefetch hints for JavaScript resources per page.", + description="Bar chart showing the distribution of prefetch hints per page. A median desktop page loads 2 JavaScript resources with the prefetch hint.", + chart_url="https://docs.google.com/spreadsheets/d/e/2PACX-1vTpHzC_cMZYj2VLzQ4ODK3uvZkNBXtwdAZriZaBwjLjUM1SGwwmJs9rv8T6OtNdXox29PQ34CasUUwc/pubchart?oid=1265752379&format=interactive", sheets_gid="1107832831", sql_file="resource-hints-prefetch-preload-percentage.sql" ) }} -A median desktop page loads 1 JavaScript resource with the preload hint whereas 2 JavaScript resources with the prefetch hint. +The median desktop page loads one JavaScript resource with the `preload` hint and two JavaScript resources with the `prefetch` hint.
- + - + - - - - - -
ClientHint 2020 2021
Desktop`preload` 1 1
Mobile11
-
{{ figure_link(caption="Year over year comparison of the usage of preload hints per page at the median.", sheets_gid="1107832831", sql_file="resource-hints-prefetch-preload-percentage.sql") }}
-
- -
- - - - - - - - - - - - - - - - +
Client20202021
Desktop32
Mobile`prefetch` 3 2
-
{{ figure_link(caption="Year over year comparison of the usage of prefetch hints per page at the median.", sheets_gid="1107832831", sql_file="resource-hints-prefetch-preload-percentage.sql") }}
+
{{ figure_link( + caption="Year-over-year comparison of the median number of `preload` and `prefetch` hints for JavaScript resources per mobile page.", + sheets_gid="1107832831", + sql_file="resource-hints-prefetch-preload-percentage.sql" + ) }}
-Looking at the trend of the usage of preload and prefetch hints over the past years, it can be noted that the trend favors a decrease in the usage of more preload and prefetch hints per page. +While the median number of `preload` hints per mobile page has stayed the same, the number of `prefetch` hints has decreased from three to two per page. Note that at the median, these results are identical for both mobile and desktop pages. -## How is JavaScript delivered +## How is JavaScript delivered? -When sending resources over the network, it becomes important to look at an efficient way of doing it, and compressing these resources using different techniques either statically or dynamically boosts the performance and makes the process faster. +JavaScript resources can be loaded more efficiently over the network with compression and minification. In this section, we'll explore the usage of both techniques to better understand the extent to which they're being utilized effectively. ### Compression -Most of the compressed resources use either gzip compression, or brotli (br) compression, or the compression technique is just not set. +Compression is the process of reducing the file size of a resource as it gets transferred over the network. This can be an effective way to improve the download times of JavaScript resources, which are highly compressible. For example, the `almanac.js` script loaded on this page is 28 KB, but only 9 KB over the wire thanks to compression. You can learn more about the ways resources are compressed across the web in the [Compression](./compression) chapter. {{ figure_markup( image="compression-requests.png", - caption="Compression methods usage percentage by request.", - description="Bar chart showing the percent of the usage of compression methods. 55% of JS requests apply the gzip compression on mobile, 31% apply br compression, and 14% do not apply any compression method.", + caption="Adoption of the methods for compressing JavaScript resources.", + description="Bar chart showing the percent of the usage of compression methods. 55% of JavaScript requests apply the gzip compression on mobile, 31% apply br compression, and 14% do not apply any compression method.", chart_url="https://docs.google.com/spreadsheets/d/e/2PACX-1vTpHzC_cMZYj2VLzQ4ODK3uvZkNBXtwdAZriZaBwjLjUM1SGwwmJs9rv8T6OtNdXox29PQ34CasUUwc/pubchart?oid=1306457429&format=interactive", sheets_gid="1182320606", sql_file="compression_method.sql" ) }} -It is noted that 55.4% of mobile JS requests apply the gzip compression method, whereas 30.8% of requests from mobile devices have the brotli compression applied. +Most JavaScript resources are either compressed using Gzip, Brotli (br), or not compressed at all (not set). 55.4% of mobile JavaScript resources use Gzip, whereas 30.8% of resources are compressed with Brotli. -An interesting observation here is the trend of the usage of these compression methods as [compared to the previous year](../2019/javascript#fig-10). The usage of gzip has gone down by almost 10% and brotli has increased by 11%. The trend definitely indicates the focus on smaller size files with more levels of compression that Brotli provides as compared to gzip. +Interestingly, compared to [the state of JavaScript compression in 2019](../2019/javascript#fig-10), Gzip has gone down by almost 10 percentage points and Brotli has increased by 16 percentage points. The trend illustrates the shift to focus on smaller size files with higher levels of compression that Brotli provides as compared to Gzip. -With such comparable data, let us actually check if the percentage difference is somehow impacted by the resources being first-party or third-party. +To help explain this change, we analyzed the compression methods of first and third-party resources. {{ figure_markup( image="compression-first-third-party.png", - caption="Compression methods for first party vs third party.", + caption="Adoption of the methods for compressing first and third-party JavaScript resources on mobile pages.", description="Bar chart showing the percent of the usage of compression methods for first-party and third-party. 59% of third-party JS requests apply the gzip compression on mobile, 30% third-party requests apply br compression, and 11% third-party scripts do not apply any compression method.", chart_url="https://docs.google.com/spreadsheets/d/e/2PACX-1vTpHzC_cMZYj2VLzQ4ODK3uvZkNBXtwdAZriZaBwjLjUM1SGwwmJs9rv8T6OtNdXox29PQ34CasUUwc/pubchart?oid=1269923134&format=interactive", sheets_gid="821396474", @@ -392,44 +381,45 @@ With such comparable data, let us actually check if the percentage difference is ) }} -59.1% of third-party scripts are gzipped and 29.6% are brotli compressed. Looking at first-party scripts, these are 51.7% with gzip compression but only 32.0% with brotli. There are still 11% of third-party scripts that do not have any compression method defined for the resources. +59.1% of third-party scripts on mobile pages are gzipped and 29.6% are compressed with Brotli. Looking at first-party scripts, these are 51.7% with Gzip compression but only 32.0% with Brotli. There are still 11.3% of third-party scripts that do not have any compression method defined. {{ figure_markup( image="uncompressed-first-third-party.png", caption="Uncompressed resources for first party vs third party.", - description="Bar chart showing the percent of the uncompressed resources for first party vs third party. 90% of uncompressed third party JS resources are < 5 KB, and uncompressed first-party resources less than 10kb are only 8%.", + description="Bar chart showing the percent of the uncompressed resources for first party vs third party. 90% of uncompressed third party JavaScript resources are < 5 KB, and uncompressed first-party resources less than 10kb are only 8%.", chart_url="https://docs.google.com/spreadsheets/d/e/2PACX-1vTpHzC_cMZYj2VLzQ4ODK3uvZkNBXtwdAZriZaBwjLjUM1SGwwmJs9rv8T6OtNdXox29PQ34CasUUwc/pubchart?oid=1043138204&format=interactive", sheets_gid="1942216454", sql_file="compression_none_by_bytes.sql" ) }} -The way preloading all our requests negates the impact of the resource hint, a similar problem occurs when we try to optimize and compress a resource that doesn't need compression and is already small. It is observed that 90% of uncompressed third party JS resources are less than 5 KB, though first-party requests trail a bit. +90% of uncompressed third-party JavaScript resources are less than 5 KB, though first-party requests trail a bit. This may help explain why so many JavaScript resources go uncompressed. Due to the diminishing returns of compressing small resources, a small script may cost more in terms of the resource consumption of server-side compression and client-side decompression than the performance benefits of saving a few bytes over the network. ### Minification -When working towards reducing the script parsing time, we tend to focus on all the opportunities to make our code smaller and more efficient. One such idea is to minify the files and bring down the bytes usage. +While compression only changes the transfer size of JavaScript resources over the network, minification actually makes the code itself smaller and more efficient. This not only helps to reduce the load time of the script but also the amount of time the client spends parsing the script. -The Lighthouse report also highlights the unminified JS being used and lists these unminified files and the opportunities to save. +The unminified JavaScript Lighthouse audit highlights the opportunities of minification. {{ figure_markup( image="unminified-js-audit-scores.png", - caption="Percentage distribution of unminified JS audit scores.", - description='Bar chart showing the percentage distribution of unminified JS audit scores. 67% of mobile pages have an "unminified JS" score between 0.9 and 1.0', + caption="Distribution of unminified JavaScript audit scores.", + description='Bar chart showing the percentage distribution of unminified JS audit scores. 67% of mobile pages have an "unminified JavaScript" score between 0.9 and 1.0', chart_url="https://docs.google.com/spreadsheets/d/e/2PACX-1vTpHzC_cMZYj2VLzQ4ODK3uvZkNBXtwdAZriZaBwjLjUM1SGwwmJs9rv8T6OtNdXox29PQ34CasUUwc/pubchart?oid=1572896641&format=interactive", sheets_gid="1539653841", sql_file="lighthouse_unminified_js.sql" ) }} -Here, 0.00 represents the worst score whereas 1.00 represents the best score. It is noted that 67.1% of mobile pages have an audit score between 0.9 and 1.0, meaning that there are still more than 30% of mobile pages with a unminified JS score of less than 0.9 with less or no minification. -This, as compared to the results from the last year(2020), shows a regression of 10% in mobile pages with "unminified JS" score between 0.9 and 1.0. +Here, 0.00 represents the worst score whereas 1.00 represents the best score. 67.1% of mobile pages have an audit score between 0.9 and 1.0. That means there are still more than 30% of pages that have an unminified JavaScript score worse than 0.9 and could make better use of code minification. Compared to the results from the [2020 edition](../2020/javascript#fig-16), the percent of mobile pages with an "unminified JS" score between 0.9 and 1.0 fell by 10 points. + +{# TODO: Rick is checking with the Lighthouse team to see if there were any major scoring algorithm changes to this audit in the past year. That would be a benign explanation for the 10 point drop in good scores. This change is really surprising, good find! #} -To understand the reason for the less score this year, let's dive deeper to look at how many bytes per page are unminified. +To understand the reason for the worse scores this year, let's dive deeper to look at how many bytes per page are unminified. {{ figure_markup( image="unminified-js-bytes.png", - caption="Percentage distribution of unminified JS bytes per page.", + caption="Distribution of the amount of unminified JavaScript per page, in KB.", description="Bar chart showing the percentage distribution of unminified JS bytes per page. 57% of mobile pages have 0 KB of unminified JS whereas 18% of those have 0-10 kilobytes of unminified JS.", chart_url="https://docs.google.com/spreadsheets/d/e/2PACX-1vTpHzC_cMZYj2VLzQ4ODK3uvZkNBXtwdAZriZaBwjLjUM1SGwwmJs9rv8T6OtNdXox29PQ34CasUUwc/pubchart?oid=1069289665&format=interactive", sheets_gid="1511556407", @@ -437,50 +427,57 @@ To understand the reason for the less score this year, let's dive deeper to look ) }} -And it is found that 57.4% of mobile pages have 0 KB of unminified JavaScript. This indicates that more than 50% of mobile pages have 100% minification with 0KBs of unminified JavaScript found. On the other hand, less than 20% of mobile pages have 0-10 kilobytes of unminified JavaScript with scope for better minification score. The rest have less to no minification and these are the pages that fall in category of bad "unminified JS" audit score that we saw in the previous chart. - -The first-party vs. third-party analysis in this case shows that 82% of the average mobile pages' unminified JS bytes actually come from first-party scripts. This indicates that the third-party scripts are definitely doing better than the first-party ones in the case of minification. +57.4% of mobile pages have 0 KB of unminified JavaScript as reported by the Lighthouse audit. 17.9% of mobile pages have between 0 and 10 KB of unminified JavaScript. The rest of the pages have an increasing number of unminified JavaScript bytes and correspond to those having poor "unminified JavaScript" audit scores in the previous chart. {{ figure_markup( image="average-unminified-js-bytes.png", - caption="Average distribution of unminified JS Bytes.", - description="Pie chart showing the average distribution of unminified JS bytes. 82% of the average mobile page's unminified JS bytes actually come from first-party scripts.", + caption="Average distribution of unminified JavaScript bytes.", + description="Pie chart showing the average distribution of unminified JavaScript bytes. 82% of the average mobile page's unminified JavaScript bytes actually come from first-party scripts.", chart_url="https://docs.google.com/spreadsheets/d/e/2PACX-1vTpHzC_cMZYj2VLzQ4ODK3uvZkNBXtwdAZriZaBwjLjUM1SGwwmJs9rv8T6OtNdXox29PQ34CasUUwc/pubchart?oid=1119550643&format=interactive", sheets_gid="127709080", sql_file="lighthouse_unminified_js_by_3p.sql" ) }} +When we segmented the unminified JavaScript resources by host, we found that 82.0% of the average mobile page's unminified JavaScript bytes actually come from first-party scripts. + ### Source maps -Source maps are files sent along with the JavaScript resource files to let the browser map the minified or compressed file to their original version of that file. This is how source maps help us in debugging in the production environment. +[Source maps](https://developer.mozilla.org/en-US/docs/Tools/Debugger/How_to/Use_a_source_map) are hints sent along with JavaScript resources that allow the browser to map the minified resource back to their source code. This is especially helpful to web developers for debugging in a production environment. {{ figure_markup( content="0.1%", - caption="Mobile pages using the sourcemap header.", + caption="Percent of mobile pages that use the `SourceMap` header.", classes="big-number", sheets_gid="1186092564", sql_file="sourcemap_header.sql" ) }} -It is found that only 0.1% of mobile pages use the SourceMap response header on script resources. One reason for this extremely small percentage could be that not many sites choose to put their original source code in production through the source map. +Only 0.1% of mobile pages use the [`SourceMap`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/SourceMap) response header on script resources. One reason for this extremely small percentage could be that not many sites choose to put their original source code in production through the source map. -An analysis of how many sites actually send the source map header on their first-party or third-party scripts shows that 0.1% of the JS requests that include a SourceMap header on mobile are for first-party scripts. +{{ figure_markup( + content="98.0%", + caption="Percent of JavaScript resources on mobile pages using the `SourceMap` header that are first-party resources.", + classes="big-number", + sheets_gid="2057978707", + sql_file="sourcemap_header.sql" + ) +}} -{# TODO That seems surprising to me. I would have though it was more common to be from third-parties since the likes of query and Bootstrap include source map files. Any thoughts on why this is the reverse? Or am I just wrong about this expectation? Nishu - These results are confusing to me. I reran the query and didn't find 98% anywhere, so I have removed the figure but simply put 0.1% as first-party results for mobile. If you'd like to have a look - here's the data - https://docs.google.com/spreadsheets/d/1zU9rHpI3nC6jTz3xgN6w13afW7x34xAKBh2IPH-lVxk/edit#gid=2057978707 #} +98.0% of the `SourceMap` usage on JavaScript resources can be attributed to first-parties. Only 2.0% of scripts with the header on mobile pages are third-party resources. ## Libraries and frameworks -Over the past years, the usage of JavaScript has increased tremendously with the adoption of many libraries and frameworks. In an effort to fasten and ease building the web, many frameworks have been introduced with each bringing features that make the app more performant. This was so prevalent that the term _framework fatigue_ came into existence. +The usage of JavaScript seems to have increased tremendously over the years, with the adoption of many new libraries and frameworks all promising their own unique improvements to the developer and user experiences. They have become so prevalent that the term framework fatigue was coined to describe developers' struggle just to keep up. In this section, we'll look at the popularity of the JavaScript libraries and frameworks in use on the web today. ### Libraries usage -To understand the usage of libraries and frameworks, HTTP Archive uses [Wappalyzer](./methodology#wappalyzer), which detects many technologies used by the web pages. +To understand the usage of libraries and frameworks, HTTP Archive uses [Wappalyzer](./methodology#wappalyzer) to detect the technologies used on a page. {{ figure_markup( image="js-libs-frameworks.png", - caption="Usage of Javascript libraries and frameworks.", + caption="Usage of JavaScript libraries and frameworks.", description="Bar chart showing the usage of Javascript libraries and frameworks. jQuery remains on top with 84% of mobile pages using it.", chart_url="https://docs.google.com/spreadsheets/d/e/2PACX-1vTpHzC_cMZYj2VLzQ4ODK3uvZkNBXtwdAZriZaBwjLjUM1SGwwmJs9rv8T6OtNdXox29PQ34CasUUwc/pubchart?oid=584103777&format=interactive", sheets_gid="1851485826", @@ -488,21 +485,31 @@ To understand the usage of libraries and frameworks, HTTP Archive uses [Wappalyz ) }} -jQuery remains on top with 84% of mobile pages containing jQuery, however React usage has jumped from 4% to 8% since last year, which is a significant increase. Also worth noticing is the usage of Isotope (uses jQuery) to 7% leading to RequireJS falling out of the picture with 2% usage. +jQuery remains the most popular library, used by a staggering 84% of mobile pages. React usage has jumped from 4% to 8% since last year, which is a significant increase. React's increase may be partially due to recent detection improvements to Wappalyzer, and may not necessarily reflect the actual change in adoption. It's also worth noting that Isotope, which uses jQuery, is found on 7% of pages, leading to RequireJS falling out of the top spots on just 2% of pages. -One might wonder why jQuery is so dominant and hasn't disappeared over time. There are two main reasons for this. First, as [highlighted over the previous years](../2019/javascript#open-source-libraries-and-frameworks), is that most of the WordPress sites use jQuery, which contributes to a major participation of jQuery. Second, even if the wide usage of jQuery is ignored for a moment, several of the other top-used JavaScript libraries still rely on jQuery in some way under the hood. +You might wonder why jQuery is still so dominant in 2021. There are two main reasons for this. First, as [highlighted over the previous years](../2019/javascript#open-source-libraries-and-frameworks), most WordPress sites use jQuery. Given that WordPress is used on nearly a third of all websites, according to the [CMS](./cms) chapter, this accounts for a huge proportion of jQuery adoption. Second, several of the other top-used JavaScript libraries still rely on jQuery in some way under the hood, contributing to indirect adoption of the library. -### Libraries used together -The adoption of JavaScript frameworks doesn't see a substantial change compared to the previous years. With the way the adoption is measured, there is a detection limitation which doesn't let Wappalyzer capture the percentage precisely. Additionally, the detection is only run on home pages which doesn't capture accurately the overall framework adoption. +{{ figure_markup( + content="3.5.1", + caption="The most popular version of jQuery.", + classes="big-number", + sheets_gid="1097559251", + sql_file="frameworks_libraries_by_version.sql" + ) +}} -It would instead be interesting to look at how the popular frameworks and libraries approach the usage of other libraries. This means how many frameworks or libraries actually rely on other libraries in production. +The most popular version of jQuery is 3.5.1, which is used by 21.3% of mobile pages. The next most popular version of jQuery is 1.12.4, at 14.4% of mobile pages. The leap to version 3.0 can be explained by a change to WordPress core in 2020, which upgraded the default version of jQuery from 1.12.4 to 3.5.1. + +### Libraries used together + +Now let's look at how the popular frameworks and libraries are used together on the same page.
- + @@ -560,16 +567,18 @@ It would instead be interesting to look at how the popular frameworks and librar
Frameworks/LibrariesFrameworks and libraries Desktop Mobile
-
{{ figure_link(caption="Javascript frameworks usage with JavaScript libraries.", sheets_gid="1934429143", sql_file="frameworks_libraries_combos.sql") }}
+
{{ figure_link( + caption="Top combinations of JavaScript frameworks and libraries used together.", + sheets_gid="1934429143", + sql_file="frameworks_libraries_combos.sql" + ) }}
-As can be observed, jQuery tops in usage with libraries and frameworks, which could also be a result of using a third-party library that includes it. This, as a result, leads to more processing time for the code that the framework or library uses with the other included libraries. - -22% of known versions of jQuery are found to be version 3.5.1. This is a big jump compared to last year's (1.12.4), which could be attributed to Wordpress which constitutes most of the participation of jQuery. +The most widely-used combination of JavaScript libraries and frameworks doesn't actually consist of multiple libraries at all! When used by itself, jQuery is found on 17.4% of mobile pages. The next most popular combination is jQuery and jQuery Migrate, which is used on 8.7% of mobile pages. In fact, all of the top 10 library and framework combinations include jQuery. ### Security vulnerabilities -Using third-party libraries comes with its own benefits and drawbacks. When using third-party libraries, one extra step for developers is to ensure the security that the library brings. Lighthouse runs this security audit for the third-party libraries used using the package name and its version. These packages can then be checked if there is any vulnerability that has been identified related to those. This is done using the Snyk's open source vulnerability database. +Using JavaScript libraries can come with its own benefits and drawbacks. When using these libraries, one drawback is that older versions may include security risks like Cross Site Scripting (XSS). [Lighthouse](./methodology#lighthouse) detects the JavaScript libraries used on a page and fails the audit if their version has any known vulnerabilities in the open-source Snyk vulnerability database. {{ figure_markup( content="63.9%", @@ -580,34 +589,40 @@ Using third-party libraries comes with its own benefits and drawbacks. When usin ) }} -As per the analysis, 63.9% of mobile pages use a JS library or framework with some security vulnerability. The number has come down from 83.5% (the previous year) and that is quite an improvement. - -It would be a great idea to now see what libraries were worked upon and have either the security vulnerability removed or the library itself is used less. +63.9% of mobile pages use a JavaScript library or framework with a known security vulnerability. For context, this number has come down from [83.5% since last year](../2020/javascript#fig-30).
- - + + - - - - - - + + - + - - + + + + + + + + + + + + + + @@ -618,73 +633,68 @@ It would be a great idea to now see what libraries were worked upon and have eit - - + + - - + + - - - - - - - - - - + + - - + + - - - + + - - + + - - + + - - + + - +
LibraryPercentageLibrary or frameworkPercent of pages
Angular0.4%
AngularJS1.0%jQuery57.6%
Bootstrap12.1%12.2%
Dojo0.5%jQuery UI10.5%
Underscore6.4%
Lo-Dash3.1%
Moment.js2.3%
GreenSock JS1.3%
Highcharts0.1%AngularJS1.0%
jQuery57.6%Mustache0.7%
jQuery Mobile 0.5%
jQuery UI10.5%
Knockout0.2%
Lo-Dash3.1%Dojo0.5%
Moment.js2.2%Angular0.4%
Mustache0.7%Vue0.2%
Underscore6.4%Knockout0.2%
Vue0.2%Highcharts0.1%
Vue (Fast path)0.2%Next.js0.0%
React0%0.0%
-
{{ figure_link(caption="List of security vulnerable libraries with their vulnerability percentage.", sheets_gid="551476210", sql_file="lighthouse_vulnerable_libraries.sql") }}
+
{{ figure_link( + caption="The percent of mobile pages found to contain a vulnerable version of a JavaScript library or framework.", + sheets_gid="551476210", + sql_file="lighthouse_vulnerable_libraries.sql" + ) }}
-The results show that jQuery is the dominant factor in the decrease of vulnerability with at least 23.3% of improvement here and some improvement in jQueryUI too. However, there is little to no difference in other libraries' security vulnerabilities. +When we segment the percent of mobile pages by library and framework, we can see that jQuery is largely responsible for the decrease in vulnerabilities. This year JavaScript vulnerabilities were found on 57.6% of pages with jQuery, compared to [80.9% last year](../2020/javascript#fig-31). As [predicted](../2020/javascript#fig-31) by [Tim Kadlec](../2020/contributors#tkadlec) in the 2020 edition of this chapter, "if we can get folks to migrate away from those outdated, vulnerable versions of jQuery, we would see the number of sites with known vulnerabilities plummet". And that's exactly what happened; WordPress migrated from jQuery version 1.12.4 to the more secure version 3.5.1, contributing to a 20 point drop in the percent of pages with known JavaScript vulnerabilities. ## How do we use JavaScript? -Now we've looked at how we get the JavaScript, what are we using it for? +Now that we've looked at how we get the JavaScript, what are we using it for? ### AJAX -A lot of JavaScript that we write is used to communicate with servers through asynchronous requests to receive information in various formats. AJAX, or asynchronous JavaScript and XML, is typically used to send and receive data in JSON, XML, HTML, text formats. +One way that JavaScript is used is to communicate with servers to asynchronously receive information in various formats. [_Asynchronous JavaScript and XML_](https://developer.mozilla.org/en-US/docs/Web/Guide/AJAX/Getting_Started) (AJAX) is typically used to send and receive data, and it supports more than just XML, including JSON, HTML, and text formats. -With multiple ways to send and receive data on the web, let's look into how many async requests are sent per page. +With multiple ways to send and receive data on the web, let's look at how many asynchronous requests are sent per page. {{ figure_markup( image="async-requests-per-page.png", - caption="The number of asynchronous requests made per page.", + caption="Distribution of the number of asynchronous requests made per page.", description="Bar chart showing the number of asynchronous requests made per page. On a desktop page, the 10th, 25th, 50th, 75th, and 90th percentiles for asynchronous requests made are: 2, 3, 4, 8, 15.", chart_url="https://docs.google.com/spreadsheets/d/e/2PACX-1vTpHzC_cMZYj2VLzQ4ODK3uvZkNBXtwdAZriZaBwjLjUM1SGwwmJs9rv8T6OtNdXox29PQ34CasUUwc/pubchart?oid=709009883&format=interactive", sheets_gid="183546956", @@ -692,26 +702,23 @@ With multiple ways to send and receive data on the web, let's look into how many ) }} -A median of 4 asynchronous requests are made per page on both mobile and desktop devices. This applies to the 50th percentile, i.e., the median desktop and mobile pages. - -If we look at the long tail and check the number of async requests for the 100th percentile, the difference in the devices is quite remarkable. The most asynchronous requests made by a desktop page is 623, which is eclipsed by the biggest mobile page, which makes 867 asynchronous requests! +The median mobile page makes 4 asynchronous requests. If we look at the long tail, the largest number of asynchronous requests on desktop pages is 623, which is eclipsed by the biggest mobile page, which makes 867 asynchronous requests! -An alternative to the asynchronous AJAX requests are the synchronous. Rather than passing a request to a callback, they block the main thread until the request completes. +An alternative to the asynchronous AJAX requests are the synchronous requests. Rather than passing a request to a callback, they block the main thread until the request completes. -However, [this practice is discouraged](https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/Synchronous_and_Asynchronous_Requests#synchronous_request) due to the potential for poor performance and user experiences, and many browsers already warn about the usage. It would be intriguing to see how many pages still use synchronous AJAX requests. +However, [this practice is discouraged](https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/Synchronous_and_Asynchronous_Requests#synchronous_request) due to the potential for poor performance and user experiences, and many browsers already warn about such usage. It would be intriguing to see how many pages still use synchronous AJAX requests. {{ figure_markup( image="usage-sync-async.png", caption="Usage of synchronous and asynchronous AJAX requests on mobile pages", description="Bar chart showing the usage of synchronous and asynchronous AJAX requests on mobile pages. The percent of synchronous requests made on mobile page are 2.5% and that of asynchronous requests are 77.6%.", chart_url="https://docs.google.com/spreadsheets/d/e/2PACX-1vTpHzC_cMZYj2VLzQ4ODK3uvZkNBXtwdAZriZaBwjLjUM1SGwwmJs9rv8T6OtNdXox29PQ34CasUUwc/pubchart?oid=189627938&format=interactive", - sheets_gid="1113974683", - sql_file="ajax_async.sql" + sheets_gid="183546956", + sql_file="ajax_request_per_page.sql" ) }} -As we see, there are still more than 2.5% of desktop and mobile pages that use the deprecated synchronous AJAX requests. -Let's look at the trend by comparing the results with the last two years. +2.5% of mobile pages use the deprecated synchronous AJAX requests. To put this into perspective, let's look at the trend by comparing the results with the last two years. {{ figure_markup( image="usage-sync-async-over-years.png", @@ -723,63 +730,15 @@ Let's look at the trend by comparing the results with the last two years. ) }} -We see there is a clear increase in the usage of asynchronous AJAX requests. However, no significant decline in the usage of synchronous AJAX requests can be seen over the years. - -An AJAX request works with different formats of data that can be sent and received. When a resource is requested, there is significant information that is sent to the server to make the results specific and clear. One such requirement is sending the relevant content types. The request sent from the browser requests a content type that gets back the data in the requested format. - -Out of the vast list of content types that can be requested, let's look at the most commonly requested content types. - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
CategoryDesktopMobile
Images45.6%44.6%
JavaScript32.3%33.3%
CSS12.6%13.2%
Fonts5.7%5.0%
JSON3.8%3.9%
-
{{ figure_link(caption="The most requested content types in AJAX.", sheets_gid="538991873", sql_file="most_requested_content_type.sql") }}
-
- -The most requested content types on mobile and desktop pages are images at 45% of requests. Images are a broad category inclusive of all the content types related to images, for example, `image/x-icon`, `image/svg+xml`, `image/webp`, `image/jpeg`, etc. - -JavaScript is the second most requested content type in AJAX requests, with 33.3% of requests on mobile pages having this content type. This represents content-types like `application/javascript`, `text/javascript`, etc. +We see that there is a clear increase in the usage of asynchronous AJAX requests. However, there isn't a significant decline in the usage of synchronous AJAX requests. Knowing the number of AJAX requests per page now, we'd also be interested in knowing the most commonly used APIs to request the data from the server. -We can broadly classify these AJAX requests into three different APIs and dig in to see the usage of each of these. The core APIs [`XMLHttpRequest`](https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest) (XHR), [`Fetch`](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch), and [`Beacon`](https://developer.mozilla.org/en-US/docs/Web/API/Beacon_API) are used commonly across websites with XHR being used primarily, however Fetch is gaining popularity and growing rapidly while Beacon has minimum usage. +We can broadly classify these AJAX requests into three different APIs and dig in to see how they're used. The core APIs [`XMLHttpRequest`](https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest) (XHR), [`Fetch`](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API/Using_Fetch), and [`Beacon`](https://developer.mozilla.org/en-US/docs/Web/API/Beacon_API) are used commonly across websites with XHR being used primarily, however `Fetch` is gaining popularity and growing rapidly while `Beacon` has low usage. {{ figure_markup( image="ajax_xhr.png", - caption="Usage of XMLHttpRequest", + caption="Distribution of the number of XMLHttpRequest requests per page.", description="Bar chart showing the usage of XMLHttpRequest per page on both desktop and mobile pages. The median mobile page makes 2 XHR requests, but at 90th percentile, makes 6 XHR requests. The 10th, 25th, 50th, 75th, and 90th percentiles for usage of XMLHttpRequest on desktop are: 0, 1, 2, 3, 6.", chart_url="https://docs.google.com/spreadsheets/d/e/2PACX-1vTpHzC_cMZYj2VLzQ4ODK3uvZkNBXtwdAZriZaBwjLjUM1SGwwmJs9rv8T6OtNdXox29PQ34CasUUwc/pubchart?oid=1164635195&format=interactive", sheets_gid="1012977216", @@ -791,7 +750,7 @@ The median mobile page makes 2 XHR requests, but at 90th percentile, makes 6 XHR {{ figure_markup( image="ajax_fetch.png", - caption="Usage of Fetch", + caption="Distribution of the number of `Fetch` requests per page.", description="Bar chart showing the usage of Fetch per page on both desktop and mobile pages. The median mobile page makes 2 Fetch requests, but at 90th percentile, makes 3 Fetch requests. The 10th, 25th, 50th, 75th, and 90th percentiles for usage of Fetch on mobile are: 0, 2, 2, 2, 3.", chart_url="https://docs.google.com/spreadsheets/d/e/2PACX-1vTpHzC_cMZYj2VLzQ4ODK3uvZkNBXtwdAZriZaBwjLjUM1SGwwmJs9rv8T6OtNdXox29PQ34CasUUwc/pubchart?oid=147712640&format=interactive", sheets_gid="1012977216", @@ -799,20 +758,21 @@ The median mobile page makes 2 XHR requests, but at 90th percentile, makes 6 XHR ) }} -In the case of the usage of the Fetch API, a median mobile page makes 2 requests per page and in the long tail, reaches 3 requests. This is reaching towards the standard XHR way of making requests and in turn, with a much cleaner approach and less boilerplate code, the overall load time will improve too. +In the case of the usage of the `Fetch` API, the median mobile page makes 2 requests, and in the long tail, reaches 3 requests. This API is becoming the standard XHR way of making requests, due in part to its cleaner approach and less boilerplate code. There may also be performance benefits to `Fetch` over the traditional XHR approach, due to the way browsers can decode large JSON payloads off the main thread. {{ figure_markup( image="ajax_beacon.png", - caption="Usage of Beacon", + caption="Distribution of the number of `Beacon` requests per page.", description="Bar chart showing the usage of Beacon per page on both desktop and mobile pages. The median mobile page makes no Beacon requests, but at 90th percentile, makes 1 Beacon request.", chart_url="https://docs.google.com/spreadsheets/d/e/2PACX-1vTpHzC_cMZYj2VLzQ4ODK3uvZkNBXtwdAZriZaBwjLjUM1SGwwmJs9rv8T6OtNdXox29PQ34CasUUwc/pubchart?oid=1483701430&format=interactive", sheets_gid="1012977216", sql_file="percentage_usage_of_different_ajax_apis.sql" ) }} -One major reason for the lower usage of Beacon could be that it is typically used for sending analytics data, especially where one wants to ensure that the request is sent even if the page might unload soon. This is, however, not guaranteed when using XHR. A good experiment for the future would be to see if some statistics could be collected around any pages using XHR for analytics data, session data, etc. -It would be interesting to also compare the adoption of XHR and Fetch over time. +`Beacon` usage is almost non-existent, with 0 requests per page until the 90th percentile, at which there's only one request per page. One possible explanation for this low adoption could be that `Beacon` is typically used for sending analytics data, especially when one wants to ensure that the request is sent even if the page might unload soon. This is, however, not guaranteed when using XHR. A good experiment for the future would be to see if some statistics could be collected around any pages using XHR for analytics data, session data, etc. + +It would be interesting to also compare the adoption of XHR and `Fetch` over time. {{ figure_markup( image="ajax-apis-per-year.png", @@ -824,61 +784,59 @@ It would be interesting to also compare the adoption of XHR and Fetch over time. ) }} -For both Fetch and XHR, the usage has tremendously increased over the years. Fetch has seen an increase of 38% on desktop pages and 48% for XHR. With a gradual increase for fetch, the focus seems to be towards cleaner requests and handling responses better. +For both `Fetch` and XHR, the usage has increased significantly over the years. `Fetch` usage on mobile pages is up 4 points and XHR is up 19 points. The gradual increase of `Fetch` adoption seems to point towards a trend of cleaner requests and better response handling. -### Web Components and the Shadow DOM +### Web Components and the shadow DOM -With the web becoming componentized, a developer building a single page application may think about a user view as a set of components. This is not only for the sake of developers building dedicated components for each feature, but also to maximize component reusability. It could be in the same app on a different view or in a completely different application. Such use cases lead to the usage of custom elements and web components in web applications. +With the web becoming [componentized](https://developer.mozilla.org/en-US/docs/Web/Web_Components), a developer building a single page application may think about a user view as a set of components. This is not only for the sake of developers building dedicated components for each feature, but also to maximize component reusability. It could be in the same app on a different view or in a completely different application. Such use cases lead to the usage of custom elements and web components in applications. It would be justified to say that with many JavaScript frameworks gaining popularity, the idea of reusability and building dedicated feature-based components has been adopted more widely. This feeds our curiosity to look into the adoption of custom elements, shadow DOM, template elements. -Custom Elements are customized elements built on top of the [`HTMLElement`](https://developer.mozilla.org/en-US/docs/Web/API/HTMLElement) class. The browser provides a `customElements` API that allows developers to define an element and register it with the browser as a custom element. - -It is noted that 2.7% of desktop pages use custom elements for one or more parts of the web page. +Custom Elements are customized elements built on top of the [`HTMLElement`](https://developer.mozilla.org/en-US/docs/Web/API/HTMLElement) API. Browsers provide a `customElements` API that allows developers to define an element and register it with the browser as a custom element. {{ figure_markup( - content="2.7%", + content="3.0%", caption="Percent of desktop pages using custom elements.", classes="big-number", sheets_gid="1991863136", - sql_file="web_components_specs.sql" + sql_file="web_components_pct.sql" ) }} -_Shadow DOM_ allows you to create a dedicated subtree in the DOM for the custom element introduced to the browser. It ensures the styles and nodes inside the element are not accessible outside the element. - -0.4% of pages use Shadow DOM specification of web components to ensure a scoped subtree for the element. +3.0% of mobile pages use custom elements for one or more parts of the web page. {{ figure_markup( content="0.4%", caption="Percent of pages using Shadow DOM.", classes="big-number", sheets_gid="1991863136", - sql_file="web_components_specs.sql" + sql_file="web_components_pct.sql" ) }} -A [`