All notable changes to this project will be documented in this file. This project adheres to Semantic Versioning.
- added
timeSpentConnectingToBrowser
,timeSpentOpeningTab
,timeSpentLoadingUrl
,timeSpentParsingPage
,timeUntilError
toreq.prerender
to allow for debugging of certain issues with the server spending too much time in different lifecycle sections - added a fix to setHeader warning by splitting headers on any line returns
- added ability to configure the chrome remote debugging port for running more than one instance of chrome on the same server
- added automatic closing of browser alert dialogs
- moved some code from responseReceived down to loadingFinished to help more accurately know when content is done downloading
- removed the deletion of some CSP headers that weren't really causing any issues
- added configurable options for pdf printing to let you override all options necessary using
this.options.pdfOptions
- fixed timeouts on redirects
- added ability to override other express options on the
.listen()
function by passing in an object now instead of just the port
- We were relying on
document.doctype
to return the full doctype string but that string changed in Chrome v72. We now parse the full doctype object directly in order to build the proper doctype and this change is backwards compatible with older Chrome versions.
- Added
domContentEventFired
so thatcheckIfPageIsDoneLoading
will wait at least fordomContentEventFired
before also waiting for all network requests to finish. This should hopefully take care of any edge cases where a page is saved too early when Chrome doesn't send new network requests during the parsing of a large .js file.
- fixed issue with creating browser tabs in a new context (to clear cookies/local storage)
LOG_REQUESTS
shows console logging from the webpage being loaded- fixed
this.options.followRedirect
typo to now bethis.options.followRedirects
- Updated Mocha to 5.2.0, Sinon to 6.1.4 and a few minor package numbers
- Added package-lock.json
- Removed a check for success in the response of
Target.disposeBrowserContext
to fix an issue with Chrome 68 removing that response object.
- Removed the
Page.addScriptToEvaluateOnNewDocument({source: 'localStorage.clear()'})
since it seemed to be causing an issue with Chrome in some cases. Going to look for a better fix here since our context should be clearing this already.
- For checking if a URL returns a redirect, we were checking to see if the request returning the redirect URL matched which failed in some cases where the encoding of the URL was different in the request. That code now checks the request ID to see if it matches the original request.
- Service worker enable/disable can be enabled/disabled on a per tab basis by setting
req.prerender.enableServiceWorker
in therequestReceived
event.
- Added ability to turn on/off services workers.
- Added
this.isBrowserConnected = false
insideserver.restartBrowser()
so the prerender server won't try to render any new requests before the browser is actually restarted. Fixes a very small edge case at scale.
- Added
localStorage.clear()
on a new page being loaded due to bug in BrowserContext local storage being cleared: https://bugs.chromium.org/p/chromium/issues/detail?id=754576
- Changed
document.getElementsByTagName('html')[0].outerHTML
todocument.firstElementChild.outerHTML
when querying pagehtml
to improve performance.
- Make sure we only call
Buffer.byteLength
on a string to fix an error in newer versions of Node
- Changed
request.loaderId
torequest.requestId
inrequestWillBeSent
due to issue with Chrome 64 changing loaderId format.
- Added ability for the prerender server to restart Chrome due to some connection issues we've been seeing after a server is running for a few hours.
- Chrome re-uses the original request ID on a redirect so we are saving off the fact that we saw a redirect to make sure we return a correct 301
- Changed dependencies from ^ to ~ to make the semver more specific
- Added removal of
<link rel="import" src="">
tags after the page is done loading to theremoveScriptTags
plugin. Imported HTML can have script tags in it, and since it's already been rendered to the page we can safely remove it when running that plugin.
- Added
if (window.customElements) customElements.forcePolyfill = true
,ShadyDOM = {force: true}
, andShadyCSS = {shimcssproperties: true}
to fix Polymer app rendering.
- Added back
res.setHeader
for plugins to use
- Set
renderType
tohtml
for non "/render" endpoint
- Added Headless Chrome as a rendering engine!
- Added new event types:
requestReceived
,tabCreated
,pageLoaded
. - Added new Prerender server option:
chromeLocation
- Added ability to request jpg and png screenshots
- Added ability to request pdf export of a page
- Added ability to request HAR file of page load times
- Removed PhantomJS and all references to it
- Removed old event types:
beforePhantomRequest
,onPhantomPageCreate
,afterPhantomRequest
,beforeSend
- Removed In Memory Cache (moved to new repo)
- Removed S3 HTML Cache (moved to new repo)
- Removed Prerender server options that are no longer needed:
workers
,iterations
,softIterations
,cookiesEnabled
,pageDoneCheckTimeout
,resourceDownloadTimeout
,jsTimeout
,noJsExecutionTimeout
,evaluateJavascriptCheckTimeout
See the Readme.me for in depth descriptions of all of the new changes!
- Whoops. Make sure
shouldEncodeURLBeforeBrowserFetch
defaults to true.
- Added
shouldEncodeURLBeforeBrowserFetch
to allow projects that use prerender to determine whether they want to callencodeURI
on the URL before fetching it in PhantomJS. Useful for some URLs that might have encoded slashes in them, since encoding them further would cause incorrect behavior.
- Fixed issue where PhantomJS crashed and then disposing caused bad phantomjs state
- Bumped all dependency versions to latest
- Added ability for cluster master to kill last known phantomjs pid for a worker if the worker dies (preventing orphaned phantomjs instances)
- Better terminating for cluster workers
- Added NUM_SOFT_ITERATIONS to try to kill phantomjs and reclaim memory when no requests are in flight. This should be set to a low number (1-10) so that PhantomJS can be restarted often when it isn't doing anything else. NUM_ITERATIONS should still be set to something like 40-50 to make sure to force kill PhantomJS even if a request is in flight.
- Added clearing of memory cache to prevent PhantomJS from returning a 304
- Fixed issue where prerender-status-code set to
200
was causing the page to skip being cached - Fixed an issue where we weren't using the correct pid when trying to force kill PhantomJS.
- Moved clearing of memory cache and local storage to before the page loads instead of after. This will prevent edge cases that could cause a 304.
- Fixed issue where S3HtmlCache was calling next() before finishing saving to the cache
- Fixed issue where we were calling
hasOwnProperty
on aquerystring
that no longer had Object on it's prototype chain
- Fixed issue where a webpage calling window.close would cause Prerender to be unable to shutdown PhantomJS properly
- S3 cache plugin was incorrectly saving non-200 status code responses to the cache
- preserve phantom arguments when server is restarting
- use default when phantomArguments is empty
- prevent multiple phantomjs instances from being started with low number of iterations
- try to check to see if phantomjs has actually been disposed. if not, force kill it.
- added engines to package.json and fixed possible bug in checking options passed in
- prevent weird hangup on error setting a header with a newline
- make sure we catch any errors thrown from phridge and continue
- kill workers (and phantomjs) on SIGTERM
- Changed
PHANTOM_CLUSTER_NUM_WORKERS
toPRERENDER_NUM_WORKERS
in server.js - Changed
PHANTOM_WORKER_ITERATIONS
toPRERENDER_NUM_ITERATIONS
in server.js - Switched from
phantomjs-node
bridge tophridge
- All Prerender plugins that access PhantomJS need to be rewritten to support new phridge integration. For example, change this:
req.prerender.page.set('onConsoleMessage', function(msg) {
console.log(msg);
});
to this:
req.prerender.page.run(function() {
this.onConsoleMessage = function(msg) {
console.log(msg);
};
});
Please see phridge for more info on how to interact with PhantomJS through phridge
.
###Removed
- Removed
PHANTOM_CLUSTER_BASE_PORT
sincephridge
doesn't start a webserve to talk to PhantomJS, so it's no longer needed. - Removed
PHANTOM_CLUSTER_MESSAGE_TIMEOUT
sincephridge
doesn't start a webserve to talk to PhantomJS, so it's no longer needed.