March 21, 2021

Extracting an ESM module from a Deno script

This is another followup to my recent post about Getting Started with Deno.

I thought it would make sense to extract the crawler code into its own ESM module so that it can also be used with Node.js or in the browser.

The resulting API is a bit ugly because it expects parse5 and fetch as parameters, but it works .

/**
 * @param {URL} rootURL
 * @param {boolean} noRecurse
 * @param {boolean} quiet
 * @param {function} parse5 - transitive dependency
 * @param {function} fetch - native or npm package
 * @param {Object} fetchOpts options passed to fetch - optional
 * @returns {Object} map of url -> { url, status, in, [error] }
 */
export default async function scanurl(rootURL, noRecurse, quiet, parse5, fetch, fetchOpts) {

Calling the ESM module from the browser

You can try running the module from inside your own browser at https://deno-hello.jldec.me/.

The page shows how to import the module from an inline <script type="module">.

<script type="module" id="code">
import scanurl from './scanurl.mjs';
import parse5 from 'https://cdn.skypack.dev/parse5';
...
</script>

Note that the usual browser CORS restrictions also apply to ESM modules, and to fetch() calls. In this case 'scanurl' is imported using a relative path on the same origin, and 'parse5' is imported using https://www.skypack.dev/.

Using the scanode ESM module with Node

I have published scanode as a package on npm. If you have Node, you can run it with 'npx' or install it using 'npm install'.

$ npx scanode https://jldec.me
npx: installed 3 in 0.987s
parsing /
...
14 pages scanned.
🎉 no broken links found.

You can also call the module API from your own code as in node_example/test-scan.js.

import fetch from 'node-fetch';
import parse5 from 'parse5';
import scanode from 'scanode';

const result = await scanode(
  new URL('https://jldec.me'),
  false, // noRecurse
  false, // quiet
  parse5,
  fetch
);

Notice the imports for 'parse5' and 'node-fetch'. These are included as dependencies in the package.json for scanode.

{
  "name": "scanode",
  "version": "2.0.1",
  "description": "ESM module - crawls a website, validating that all the links on the site which point to the same orgin can be fetched.",
  "main": "scanurl.mjs",
  "bin": {
    "scanode": "./bin/scanode.mjs"
  },
  "dependencies": {
    "node-fetch": "^2.6.1",
    "parse5": "^6.0.1"
  }
  ...

So what's wrong with this picture?

As discussed before, the NPM ecosystem predates ESM modules, so the two worlds don't play very nicely together. Node.js programs cannot easily load ESM modules which are not in NPM. Meanwhile, browsers know nothing about package.json or the node_modules directory.

When ESM modules depend on other modules, they use 'import' statements with a URL or relative path. Node.js expects those sub-modules to be referenced by their NPM package names.

The result is that modules which depend on other modules are not portable between the two worlds, without an additional transformation step, or maybe an import map.

And this is why, for now, the API above expects the parse5 module dependency as a parameter.

The big question is whether the NPM ecosystem will evolve to support nested ESM modules, or whether some other organization with a workable trust model will emerge to replace it.

Where there's a problem, there's an opportunity!

🚀

To leave a comment
please visit dev.to/jldec

debug

user: anonymous

{
  "path": "/blog/extracting-an-esm-module-from-a-deno-script",
  "attrs": {
    "title": "Extracting an ESM module from a Deno script",
    "splash": {
      "image": "/images/persewide.jpg"
    },
    "date": "2021-03-21",
    "layout": "BlogPostLayout",
    "excerpt": "How to extract an ESM module so that it can also be used with Node.js or in the browser.\n\nWill the NPM ecosystem evolve to support nested ESM modules, or will some other organization, with a workable trust model, emerge to replace it?\n"
  },
  "md": "# Extracting an ESM module from a Deno script\n\nThis is another followup to my recent post about [Getting Started with Deno](getting-started-with-deno).\n\nI thought it would make sense to extract the crawler code into its own [ESM module](migrating-from-cjs-to-esm) so that it can also be used with Node.js or in the browser.\n\nThe resulting [API](https://github.com/jldec/deno-hello/blob/main/scanurl.mjs#L18) is a bit ugly because it expects parse5 and fetch as parameters, but it works  .\n\n```js\n/**\n * @param {URL} rootURL\n * @param {boolean} noRecurse\n * @param {boolean} quiet\n * @param {function} parse5 - transitive dependency\n * @param {function} fetch - native or npm package\n * @param {Object} fetchOpts options passed to fetch - optional\n * @returns {Object} map of url -> { url, status, in, [error] }\n */\nexport default async function scanurl(rootURL, noRecurse, quiet, parse5, fetch, fetchOpts) {\n```\n\n## Calling the ESM module from the browser\n\nYou can try running the module from inside your own browser at https://deno-hello.jldec.me/.\n\n[![Screenshot of https://deno-hello.jldec.me](/images/deno-hello.jldec.me.png)](https://deno-hello.jldec.me/)\n\nThe page shows how to import the module from an inline `<script type=\"module\">`.\n\n```html\n<script type=\"module\" id=\"code\">\nimport scanurl from './scanurl.mjs';\nimport parse5 from 'https://cdn.skypack.dev/parse5';\n...\n</script>\n```\n\nNote that the usual browser CORS restrictions also apply to ESM modules, and to fetch() calls. In this case 'scanurl' is imported using a relative path on the same origin, and 'parse5' is imported using https://www.skypack.dev/.\n\n## Using the scanode ESM module with Node\n\nI have published [scanode](https://www.npmjs.com/package/scanode) as a package on npm. If you have Node, you can run it with 'npx' or install it using 'npm install'.\n\n```\n$ npx scanode https://jldec.me\nnpx: installed 3 in 0.987s\nparsing /\n...\n14 pages scanned.\n🎉 no broken links found.\n```\n\nYou can also call the module API from your own code as in [node_example/test-scan.js](https://github.com/jldec/deno-hello/blob/main/node_example/test-scan.js).\n\n```js\nimport fetch from 'node-fetch';\nimport parse5 from 'parse5';\nimport scanode from 'scanode';\n\nconst result = await scanode(\n  new URL('https://jldec.me'),\n  false, // noRecurse\n  false, // quiet\n  parse5,\n  fetch\n);\n```\n\nNotice the imports for 'parse5' and 'node-fetch'. These are included as dependencies in the [package.json](https://github.com/jldec/deno-hello/blob/main/package.json#L9) for scanode.\n\n```json\n{\n  \"name\": \"scanode\",\n  \"version\": \"2.0.1\",\n  \"description\": \"ESM module - crawls a website, validating that all the links on the site which point to the same orgin can be fetched.\",\n  \"main\": \"scanurl.mjs\",\n  \"bin\": {\n    \"scanode\": \"./bin/scanode.mjs\"\n  },\n  \"dependencies\": {\n    \"node-fetch\": \"^2.6.1\",\n    \"parse5\": \"^6.0.1\"\n  }\n  ...\n```\n\n## So what's wrong with this picture?\n\nAs discussed [before](migrating-from-cjs-to-esm), the NPM ecosystem predates ESM modules, so the two worlds don't play very nicely together. Node.js programs cannot easily load ESM modules which are not in NPM. Meanwhile, browsers know nothing about package.json or the node_modules directory.\n\nWhen ESM modules depend on other modules, they use 'import' statements with a URL or relative path. Node.js expects those sub-modules to be referenced by their NPM package names.\n\nThe result is that modules which depend on other modules are not portable between the two worlds, without an additional transformation step, or maybe an [import map](https://caniuse.com/import-maps).\n\nAnd this is why, for now, the API above expects the `parse5` module dependency as a parameter.\n\n> The big question is whether the NPM ecosystem will evolve to support nested ESM modules, or whether some other organization with a workable trust model will emerge to replace it.\n\nWhere there's a problem, there's an opportunity!\n\n# 🚀\n\n_To leave a comment  \nplease visit [dev.to/jldec](https://dev.to/jldec/extracting-an-esm-module-from-a-deno-script-28il)_\n\n\n\n\n\n\n\n\n\n\n\n",
  "html": "<h1>Extracting an ESM module from a Deno script</h1>\n<p>This is another followup to my recent post about <a href=\"getting-started-with-deno\">Getting Started with Deno</a>.</p>\n<p>I thought it would make sense to extract the crawler code into its own <a href=\"migrating-from-cjs-to-esm\">ESM module</a> so that it can also be used with Node.js or in the browser.</p>\n<p>The resulting <a href=\"https://github.com/jldec/deno-hello/blob/main/scanurl.mjs#L18\">API</a> is a bit ugly because it expects parse5 and fetch as parameters, but it works  .</p>\n<pre><code class=\"language-js\">/**\n * @param {URL} rootURL\n * @param {boolean} noRecurse\n * @param {boolean} quiet\n * @param {function} parse5 - transitive dependency\n * @param {function} fetch - native or npm package\n * @param {Object} fetchOpts options passed to fetch - optional\n * @returns {Object} map of url -&gt; { url, status, in, [error] }\n */\nexport default async function scanurl(rootURL, noRecurse, quiet, parse5, fetch, fetchOpts) {\n</code></pre>\n<h2>Calling the ESM module from the browser</h2>\n<p>You can try running the module from inside your own browser at <a href=\"https://deno-hello.jldec.me/\">https://deno-hello.jldec.me/</a>.</p>\n<p><a href=\"https://deno-hello.jldec.me/\"><img src=\"/images/deno-hello.jldec.me.png\" alt=\"Screenshot of https://deno-hello.jldec.me\"></a></p>\n<p>The page shows how to import the module from an inline <code>&lt;script type=&quot;module&quot;&gt;</code>.</p>\n<pre><code class=\"language-html\">&lt;script type=&quot;module&quot; id=&quot;code&quot;&gt;\nimport scanurl from './scanurl.mjs';\nimport parse5 from 'https://cdn.skypack.dev/parse5';\n...\n&lt;/script&gt;\n</code></pre>\n<p>Note that the usual browser CORS restrictions also apply to ESM modules, and to fetch() calls. In this case 'scanurl' is imported using a relative path on the same origin, and 'parse5' is imported using <a href=\"https://www.skypack.dev/\">https://www.skypack.dev/</a>.</p>\n<h2>Using the scanode ESM module with Node</h2>\n<p>I have published <a href=\"https://www.npmjs.com/package/scanode\">scanode</a> as a package on npm. If you have Node, you can run it with 'npx' or install it using 'npm install'.</p>\n<pre><code>$ npx scanode https://jldec.me\nnpx: installed 3 in 0.987s\nparsing /\n...\n14 pages scanned.\n🎉 no broken links found.\n</code></pre>\n<p>You can also call the module API from your own code as in <a href=\"https://github.com/jldec/deno-hello/blob/main/node_example/test-scan.js\">node_example/test-scan.js</a>.</p>\n<pre><code class=\"language-js\">import fetch from 'node-fetch';\nimport parse5 from 'parse5';\nimport scanode from 'scanode';\n\nconst result = await scanode(\n  new URL('https://jldec.me'),\n  false, // noRecurse\n  false, // quiet\n  parse5,\n  fetch\n);\n</code></pre>\n<p>Notice the imports for 'parse5' and 'node-fetch'. These are included as dependencies in the <a href=\"https://github.com/jldec/deno-hello/blob/main/package.json#L9\">package.json</a> for scanode.</p>\n<pre><code class=\"language-json\">{\n  &quot;name&quot;: &quot;scanode&quot;,\n  &quot;version&quot;: &quot;2.0.1&quot;,\n  &quot;description&quot;: &quot;ESM module - crawls a website, validating that all the links on the site which point to the same orgin can be fetched.&quot;,\n  &quot;main&quot;: &quot;scanurl.mjs&quot;,\n  &quot;bin&quot;: {\n    &quot;scanode&quot;: &quot;./bin/scanode.mjs&quot;\n  },\n  &quot;dependencies&quot;: {\n    &quot;node-fetch&quot;: &quot;^2.6.1&quot;,\n    &quot;parse5&quot;: &quot;^6.0.1&quot;\n  }\n  ...\n</code></pre>\n<h2>So what's wrong with this picture?</h2>\n<p>As discussed <a href=\"migrating-from-cjs-to-esm\">before</a>, the NPM ecosystem predates ESM modules, so the two worlds don't play very nicely together. Node.js programs cannot easily load ESM modules which are not in NPM. Meanwhile, browsers know nothing about package.json or the node_modules directory.</p>\n<p>When ESM modules depend on other modules, they use 'import' statements with a URL or relative path. Node.js expects those sub-modules to be referenced by their NPM package names.</p>\n<p>The result is that modules which depend on other modules are not portable between the two worlds, without an additional transformation step, or maybe an <a href=\"https://caniuse.com/import-maps\">import map</a>.</p>\n<p>And this is why, for now, the API above expects the <code>parse5</code> module dependency as a parameter.</p>\n<blockquote>\n<p>The big question is whether the NPM ecosystem will evolve to support nested ESM modules, or whether some other organization with a workable trust model will emerge to replace it.</p>\n</blockquote>\n<p>Where there's a problem, there's an opportunity!</p>\n<h1>🚀</h1>\n<p><em>To leave a comment<br>\nplease visit <a href=\"https://dev.to/jldec/extracting-an-esm-module-from-a-deno-script-28il\">dev.to/jldec</a></em></p>\n"
}