Network requests
editNothing is fetched unless the browsing context has a DocumentLoader. With a loader, the browsing context loads the documents of the navigations and of the iframe elements, the style sheets (<link rel=stylesheet>, @import), the images, the fonts, and, with a script engine, the scripts and the requests of fetch(), XMLHttpRequest, WebSocket and EventSource.
Load pages over HTTP
DocumentLoader.Create() returns a loader that sends HTTP requests:
using var context = new BrowsingContext(new BrowsingContextOptions
{
DocumentLoader = DocumentLoader.Create(), // HTTP GET requests with a shared HttpClient (or pass your own)
CookieJar = new CookieJar(), // optional: the cookies (RFC 6265bis, document.cookie, cookieStore)
});
context.Navigate("https://example.com/");
await context.EventLoop.RunUntilIdleAsync(); // runs the tasks until the document and its resources are loaded
var document = context.Document;- Without an argument, the loader uses a shared
HttpClientthat decompresses the responses. Pass your ownHttpClienttoDocumentLoader.Create(httpClient)to set a proxy, a timeout or a handler; it should not store cookies itself (UseCookies = false) when the browsing context has a CookieJar. - The redirects are followed (up to 20), the
Refererheader is sent as a browser sends it, and the cookies of the CookieJar are sent and stored. - Only the
http:andhttps:URLs are requested. Thedata:andblob:URLs are handled by the library. - There is no prompt for the HTTP authentication (
401withWWW-Authenticate): when the URL of the request has credentials (https://user:password@example.com/), they answer the challenge, as in the browsers, also for the subresources (images, scripts...). Once accepted, they are used for the other requests to the same origin.
Upgrade the navigations to HTTPS
A browser that implements HTTPS upgrades (a proposal for the Fetch standard, https://github.com/whatwg/fetch/pull/1655) requests the https: URL of an http: link, and falls back to the http: URL when the https: request fails. Set HttpsUpgrades to do the same:
using var context = new BrowsingContext(new BrowsingContextOptions
{
DocumentLoader = DocumentLoader.Create(),
HttpsUpgrades = new HttpsUpgradeOptions
{
FallbackTimeout = TimeSpan.FromSeconds(3), // optional: do not wait for the timeout of the network
},
});
context.Navigate("http://example.com/"); // requests https://example.com/, then http://example.com/ if it fails
await context.EventLoop.RunUntilIdleAsync();- Only the
GETnavigations of the top-level browsing contexts are upgraded, including thehttp:URLs of their redirects. The navigations of theiframeelements, the form submissions that post data and the subresources keep their URL (theupgrade-insecure-requestsdirective of a page and the mixed content checks upgrade or block the subresources). - The navigation falls back to the
http:URL when the loader returns no response or throws (the connection or the certificate is refused), when FallbackTimeout expires before the response, or when thehttps:URL redirects back to thehttp:URL. An HTTP error (404,500) is a response: it does not fall back. The host of a failed upgrade is not upgraded again for 7 days by the browsing contexts that share the options. - The referrer is computed for the
https:URL, so ano-referrer-when-downgradepolicy sends it, and the document has thehttps:URL. - The IP addresses,
localhost, the hosts of a private network (intranet,printer.local,*.test, the top-level domains that are not in the public suffix list) and the ports other than 80 are not upgraded. UpgradeNonUniqueHosts and UpgradeNonDefaultPorts upgrade them too, keeping the port: use them for a test server that serves HTTPS on the port of thehttp:URL.
The option is off by default. The library loads the URLs your application gives it, often from local or test servers, and an upgrade sends a request you did not ask for and changes the URL of the document. The feature is not part of the Fetch standard yet, and the browsers that implement it do not all enable it in every mode.
Protect your network from untrusted pages
DocumentLoader.Create() requests any URL that a page navigates to or references. With an untrusted page, the requests can reach the services of your private network (server-side request forgery). Use a loader that filters the requests:
internal sealed class SameHostLoader(string host) : DocumentLoader
{
private static readonly DocumentLoader Http = DocumentLoader.Create();
public override ValueTask<DocumentResource?> LoadAsync(DocumentRequest request, CancellationToken cancellationToken)
{
if (new Uri(request.Url).Host != host)
return ValueTask.FromResult<DocumentResource?>(null); // network error
return Http.LoadAsync(request, cancellationToken);
}
}A null result is a network error, as is an exception thrown by the loader. DocumentRequest describes the request: its Url, Method, Headers and Body, its Destination (a document, a style sheet, an image, a script...), the Origin of the document that makes it (null when your application navigates), its Referrer, and the BrowsingContext it is for.
The URL keeps the fragment of the reference: <use href="icons.svg#home"> requests https://example.com/icons.svg#home, and <iframe src="frame.html#intro"> requests https://example.com/frame.html#intro. An HTTP request does not send it, so remove it when you look up a resource by URL.
Serve pages from memory or from the disk
A loader can return anything: the files of a folder, the pages of a cache, or test pages. Return a DocumentResource with the content, the status code and the headers:
/// <summary>Serves the documents and the resources from memory, by absolute URL without fragment.</summary>
private sealed class InMemoryLoader(IReadOnlyDictionary<string, (string Content, string ContentType)> files) : DocumentLoader
{
public override ValueTask<DocumentResource?> LoadAsync(DocumentRequest request, CancellationToken cancellationToken)
{
// The URL keeps its fragment (e.g. "frame.html#intro", "icons.svg#home"), which is not sent over HTTP
var url = request.Url.Split('#', 2)[0];
if (!files.TryGetValue(url, out var file))
return ValueTask.FromResult<DocumentResource?>(new DocumentResource("Not found") { StatusCode = 404 });
return ValueTask.FromResult<DocumentResource?>(new DocumentResource(file.Content)
{
Headers = [new("Content-Type", file.ContentType)],
});
}
}var loader = new InMemoryLoader(new Dictionary<string, (string Content, string ContentType)>(StringComparer.Ordinal)
{
["https://example.com/"] = ("<link rel=stylesheet href=site.css><h1>Home</h1><iframe src=/frame.html#intro></iframe>", "text/html"),
["https://example.com/site.css"] = ("h1 { color: green }", "text/css"),
["https://example.com/frame.html"] = ("<p>Inside the frame</p>", "text/html"),
});
using var context = new BrowsingContext(new BrowsingContextOptions { DocumentLoader = loader });
context.Navigate("https://example.com/");
await context.EventLoop.RunUntilIdleAsync(); // the document, its style sheet, the iframe and its document
var document = context.Document;
Assert.Equal("rgb(0, 128, 0)", context.Window.GetComputedStyle(document.QuerySelector("h1")!).Color);
var frame = (HtmlIFrameElement)document.QuerySelector("iframe")!;
Assert.Equal("Inside the frame", frame.ContentDocument!.Body!.TextContent);Responses
// A resource created from its decoded content
var page = new DocumentResource("<p>Hello</p>");
Assert.Equal("<p>Hello</p>", page.Content);
// A resource created from its bytes, decoded using the charset of the Content-Type header
var bytes = "console.log('Hello');"u8.ToArray();
var script = new DocumentResource(bytes) { Headers = [new("Content-Type", "text/javascript; charset=utf-8")] };
Assert.Null(script.Content);
Assert.Equal(bytes, script.ContentBytes?.ToArray());- The
Content-Typeheader gives the type of the resource (text/htmlwhen there is none) and its encoding. The bytes are decoded as a browser decodes them: byte order mark,charset, then, for HTML,<meta charset>and the detection of the encoding from the content. text/htmlis parsed as HTML, the XML types (application/xhtml+xml,image/svg+xml,application/xml) as XML, andtext/plainand the other text types as a text document.- As in a browser, the type of a page is sniffed from its bytes when the
Content-Typeis not a valid MIME type, and atext/plainpage whose first bytes are binary data is a binary resource (the default type of Apache is wrong for these files), unless the response hasX-Content-Type-Options: nosniff. A binary resource (application/octet-stream) does not create a document: it is a download, as a response whoseContent-Dispositionisattachment, and the page stays. - A document is displayed whatever its status (a 404 page is a page). A script or a style sheet whose status is not in the 200-299 range is a network error.
- The library reads the headers that change the document:
Content-Security-Policy,Referrer-Policy,Refresh,Link,X-Frame-Options,Content-Language,Document-Policy... The other headers are ignored. - The
upgrade-insecure-requestsdirective of aContent-Security-Policyupgrades thehttp:requests of the page tohttps:. As in Chromium, the requests tolocalhostand to the loopback addresses (127.0.0.1,[::1]) are not upgraded: a site served overhttp://localhostby a test server loads its same-origin resources (e.g. the SVG sprites of its<use>elements) from the same origin. - The
Linkheader of a page preloads (preload,modulepreload) and prefetches (prefetch) its resources. As in Firefox, itsstylesheetandiconlinks are requested, but the style sheet does not apply and the icon is not rendered. TheLinkheaders of the other responses (style sheets, images, scripts, fonts...) are ignored. - The responses are stored in an HTTP cache shared by the browsing contexts of the loader, as in a browser: a fresh response is used without a request, a stale one is revalidated with
If-None-MatchandIf-Modified-Since, and a stale response within itsstale-while-revalidatewindow is used while a request revalidates it in the background. The partial responses (206) of the range requests are stored too: a fresh stored response (complete or partial) answers the range requests whose range it contains, and a request withoutRangeheader completes a stored partial response that starts at the first byte with a request of the rest of the representation (Range: bytes=<next byte>-, with anIf-Rangeheader when the partial response has a validator). - A
421 Misdirected Requestresponse is retried once, as the browsers retry it on a new connection: the loader gets the request again. A request whose body is aReadableStreamis not retried (its body cannot be sent again). - The opaque range responses (
206or416) of the range requests (e.g. of a media element) are network errors for the no-cors requests withoutRangeheader: a service worker cannot give the partial response of an audio file, stored or put in a cache, to a script element. - A
fetch()request whose body is aReadableStream(duplex: "half") reads the stream before the request is sent to the loader: the response arrives once the body is sent, as with the half duplex of the browsers. The body of a request of another scheme thanhttp:andhttps:(e.g. adata:URL) is not read. - The icon of a top-level document (
<link rel=icon>) is requested (destinationimage, a CORS request with acrossoriginattribute), as the browsers request the icon of a tab, but it is not rendered. The element fires noloadorerrorevent, and the request has no resource timing entry. - Compression dictionaries are supported. A response of a secure URL with a
Use-As-Dictionaryheader (fetched with CORS or from the same origin) registers its body as a dictionary for the URLs of itsmatchpattern, while it would be fresh in the HTTP cache. A header whose pattern is not valid, has regular expression groups, or is not same-origin with the dictionary is ignored. The next requests of these URLs announce the best dictionary in theirAvailable-DictionaryandDictionary-IDheaders (andAccept-Encoding: dcb, dcz), and the library decodes the responses compressed with it (Content-Encoding: dcb, Brotli, ordcz, Zstandard); the loader decodes the other content codings. A response that cannot be decoded has a body that fails. The dictionaries are partitioned like the HTTP cache, and thecacheandcookiestypes ofClear-Site-Dataclear them. A dictionary is at most 16 MiB (as a body of the HTTP cache), and the dictionaries of a document loader take at most 64 MiB: the ones registered the longest ago are removed first. A<link rel="compression-dictionary">element and aLinkheader with this relation fetch a dictionary once the page is loaded. The Brotli decoder of .NET has no dictionary support: the library decodes mostdcbstreams, but a stream whose dictionary and data do not fit in its window, or whose first literals would depend on the bytes before the stream, is not supported (its body fails). - Document Policy is applied as in Chromium:
Document-Policy: sync-xhr=?0makes the synchronousXMLHttpRequestrequests fail, andDocument-Policy-Report-Onlyreports them (document-policy-violationreports). A page requires a policy from the pages of its frames with theRequire-Document-Policyheader and thepolicyattribute of itsiframeelements: the requests of the frames send it in theSec-Required-Document-Policyheader, and a frame whoseDocument-Policyheader is less strict is blocked. The library supports thesync-xhrandforce-load-at-topfeatures. Disable it with DocumentPolicyEnabled. - The
Cross-Origin-Opener-PolicyandCross-Origin-Opener-Policy-Report-Onlyheaders of a page report to the endpoint of theirreport-toparameter (an endpoint of theReporting-Endpointsheader, or of the legacyReport-Toheader): the navigations that switch, or would switch, the browsing context group of a page that shares it with other windows (navigation-to-responseandnavigation-from-responsereports, also for the redirects), and, as in Chromium, the accesses between two windows that a report-only policy would separate (access-from-coop-page-to-openeeand the otheraccess-*reports, with the name of the property). TheReportingObservers of the page that performs the access receive its report as acoop-access-violationreport. - As in Chromium, the page reports the use of deprecated features (
deprecationreports, once per page and feature: a synchronousXMLHttpRequestin a window) and the requests that the library does not honor (interventionreports: apreventDefault()call in atouchstart,touchmove,wheelormousewheellistener of the window, the document, the root element or the body that is passive because it is registered without thepassiveoption). TheReportingObservers of the page receive them, and they are sent to thedefaultendpoint of theReporting-Endpointsheader. The body gives the identifier of the feature (as in Chromium, e.g.XMLHttpRequestSynchronousInNonWorkerOutsideBeforeUnload,PreventDefaultPassive), a message and the location of the script. The non-standard members that the browsers deprecate (e.g.webkitRequestAnimationFrame(),window.fullScreen) are not exposed, so they report nothing. - The scripts, the style sheets and the preloads check the hashes of their
integrityattribute (Subresource Integrity): a resource whose body does not match is a network error (the element fireserror, the script does not run, the style sheet does not apply), and so is a cross-origin resource with hashes fetched without CORS. A response whoseUnencoded-Digestheader (sha-256,sha-512) does not match its body is a network error too, except the responses thatfetch()streams. TheIntegrity-Policyheader of a page blocks its scripts and style sheets (blocked-destinations=(script style)) that have no hash or are not CORS requests;Integrity-Policy-Report-Onlyonly reports them (integrity-violationreports, sent to theendpointsof the policy). The documents that inherit the policies of the page (itssrcdocandabout:blankframes and popups) inherit both policies too. - Signature-based Subresource Integrity is supported: an
integrityattribute (or theintegrityoption offetch()) with Ed25519 public keys (ed25519-<base64 key>) requires a valid signature of one of them, sent in theSignatureandSignature-Inputheaders (HTTP Message Signatures, with theed25519-integritytag) and covering theUnencoded-Digestheader. The requests announce the keys in theirAccept-Signatureheader. A response whose signature of this profile is not valid is a network error, whatever its request. - As in the browsers, a request whose URL has both a newline or a tab and a
<(e.g.<img src='https://evil.example/?whose quote is not closed, which captures the markup that follows it) is blocked: the dangling markup mitigation. - The
Content-DPRheader of an image gives its density: its natural size is divided by it (Content-DPR: 2displays a 100x100 image at 50x50 CSS pixels), in theimgelements (where it replaces the density of thesrcsetcandidate), the CSS images, the SVGimageelements, the video posters and the image buttons. - RemainingContentBytes streams a body that arrives over time: the HTML of a navigation is parsed as its chunks arrive, the body of
fetch()is a stream, and an asynchronousXMLHttpRequestfires itsreadystatechangeandprogressevents as the chunks arrive (at most every 50 ms; itsresponseTexthas every chunk received so far). Return the response once its headers are received: the time the loader returns it is theresponseStartof the resource timing entry, and the end of the body itsresponseEnd. - The resource timing entries (
performance.getEntriesByType("resource")) have the experimentalinitiatorUrlof Chromium (PerformanceResourceTiming.InitiatorUrl, whose .NET property reports the experimental diagnosticWEBPLATEXP0026): the style sheet that fetched a resource (its@importrules, fonts and images), the module that imports a module, else the script that started the fetch, followed through the timers, the tasks, the callbacks and the asynchronous operations it scheduled (an inline script is the page), else the page or the worker. - The navigation timing entry of a page served by the HTTP cache has the
cachedelivery type, and itstransferSizeis 0 (300 when the stored response was validated). The navigation preload request of a service worker is reported to the performance timeline of the service worker, with thenavigationinitiator type. - ReportInterimResponse reports an interim response (
100 Continue,103 Early Hints...) before the final response: its time is thefirstInterimResponseStartof the resource timing entry, and theLinkheaders of the first103response of the navigation of a page (not a frame) preload their resources, which the page uses (their timing entries have theearly-hintsinitiator type), as in Chromium.HttpClientdoes not expose the interim responses: the loader ofDocumentLoader.Create()does not report them. A loader that sends a request to another loader with a newDocumentRequestforwards them with InterimResponseHandler.
The requests are canceled when the navigation is aborted (another navigation, the removal of the iframe) and when the browsing context is disposed: pass the CancellationToken to your I/O.
Network Error Logging
Network Error Logging is disabled by default: the reports describe the requests of the user, and are sent to collectors that the sites choose. Enable it with BrowsingContextOptions.NetworkErrorLogging:
// Network Error Logging is disabled by default: give the browsing contexts a cache of the NEL policies
var networkErrorLogging = new NetworkErrorLoggingCache();
var options = new BrowsingContextOptions { NetworkErrorLogging = networkErrorLogging };- The
NELheader of a secure response ({"report_to": "group", "max_age": 86400}) sets the policy of its origin, andmax_age: 0removes it. Amax_ageabove 10 years is clamped to 10 years.report_tonames an endpoint of theReporting-Endpointsheader or a group of the legacyReport-Toheader of the same response. - Each request sent to the network (the redirects and the
304revalidations included, not the responses of the HTTP cache) to an origin with a policy generates anetwork-errorreport, sent to the endpoint of the policy:ok,http.error(4xxand5xx), or the type of the failure. The loader reports the failure with theHttpRequestErrorof theHttpRequestExceptionit throws (dns.name_not_resolved,tcp.failed,tls.failed...). Thesuccess_fraction(0 by default) andfailure_fraction(1 by default) members sample the reports. - A policy with
include_subdomainsalso applies to the subdomains of its origin, for their DNS failures only (the URL of the report is then the origin of the request). - The reports are not visible to the
ReportingObservers. The cache can be shared by several browsing contexts.
Downloads
A hyperlink with a download attribute, and a navigation to a response that is not displayed (its Content-Disposition is attachment, or its MIME type is application/octet-stream), start a download: the page stays, and DownloadStarted receives the response (Download) to save it:
var loader = new InMemoryLoader(new Dictionary<string, (string Content, string ContentType)>(StringComparer.Ordinal)
{
["https://example.com/"] = ("<a href=/export.csv download=report.csv>Export</a>", "text/html"),
["https://example.com/export.csv"] = ("id,name\n1,Ada\n", "text/csv"),
["https://example.com/archive.bin"] = ("binary", "application/octet-stream"),
});
var downloads = new List<string>();
using var context = new BrowsingContext(new BrowsingContextOptions
{
DocumentLoader = loader,
// The callback saves the response: Filename is the download attribute of the hyperlink, or null for a navigation
DownloadStarted = (_, download) => downloads.Add($"{download.Url} {download.Filename ?? "(navigation)"}: {download.Resource.Content}"),
});
context.Navigate("https://example.com/");
await context.EventLoop.RunUntilIdleAsync();
var page = context.Document;
((HtmlElement)page.QuerySelector("a")!).Click(); // a hyperlink with a download attribute
context.Window.Location.Assign("/archive.bin"); // a response that is not displayed (application/octet-stream)
await context.EventLoop.RunUntilIdleAsync();
Assert.Same(page, context.Document); // the page stays
Assert.Equal(["https://example.com/export.csv report.csv: id,name\n1,Ada\n", "https://example.com/archive.bin (navigation): binary"], downloads);- Filename is the value of the
downloadattribute (possibly empty), or null for a navigation; theContent-Dispositionheader of Resource may suggest another name. - A body that arrives over time (RemainingContentBytes) is not read by the library: enumerate it to save the rest of the file, or dispose its enumerator to cancel the download.
- Without a callback, the downloads are ignored: the hyperlinks with a
downloadattribute fetch nothing. The documents sandboxed withoutallow-downloadsdo not download, and the listeners of thenavigateevent can cancel a download.
WebSockets
The WebSocket connections of the scripts are opened by DocumentLoader.ConnectWebSocketAsync: the loader of DocumentLoader.Create() connects to the server, and a custom loader can filter the connections or return an in-memory connection. The scripts can also use WebSocketStream, the proposal that Chromium ships, whose connection is opened the same way:
- The
openedpromise gives aReadableStreamof the received messages (a string for a text message, aUint8Arrayfor a binary message) and aWritableStreamthat sends a string as a text message and aBufferSourceas a binary message. A write is fulfilled once the message is sent. - The readable stream applies backpressure: the next message is read from the network only when the page has read the previous one, so a server that sends faster than the page reads is slowed down.
close(),cancel()of the readable stream andabort()of the writable stream close the connection; theclosedpromise gives the status code and the reason of the closing handshake. A connection that cannot be established or is not closed cleanly rejects the promises with aWebSocketError, and thesignaloption aborts the opening handshake.- As a
WebSocket, the connection is closed when its document is unloaded or its worker ends.
The .NET types (WebSocketStream, WebSocketError) are experimental (WEBPLATEXP0027). A WebSocketError gives the status code and the reason of the Close frame:
// A reason without status code gives the 1000 (normal closure) status code
var error = new WebSocketError("The chat is closed", new WebSocketCloseInfo { Reason = "maintenance" });
Assert.Equal("WebSocketError", error.Name);
Assert.Equal((ushort)1000, error.CloseCode);
Assert.Equal("maintenance", error.Reason);
// The status code must be 1000 or between 3000 and 4999, and the reason at most 123 bytes in UTF-8
var invalid = Assert.Throws<DomException>(() => new WebSocketError("", new WebSocketCloseInfo { CloseCode = 1001 }));
Assert.Equal("InvalidAccessError", invalid.Name);