Searching for a Search Solution

Posted on April 20, 2023

A few months ago I sat in on a CFE.dev webinar and I was really impressed with what I saw. I made a note to come back and look closer at Pagefind, and my new blog – this blog that you’re presumably reading right now – really needed a search feature, so the “Search” box on this page (I hope it’s there) is the outcome.

Glad I Found Pagefind

So, I started looking closely at search options and considered things like Lunr, which I’ve used before, and Solr, which I both love and hate (because of its JAVA roots). Pagefind was, of course, also on that short list and it quickly solidified its position at the top of the list when I found Adding search to an Eleventy site by Mike.

Mike doesn’t explain the setup in detail, but this comment really resonated with me so I decided to follow the same path:

Weirdly, I’ve found that I seem to read the posts on this site more than I write them. I say read, but I just mean referring to the stuff I’ve written because it has left my head.

Mike does provide a link to the reference he used, namely Using PageFind with Eleventy for Search by Robb Knight. I found Robb’s guidance and experience to be VERY helpful. Thank you, Robb!

Implementing Robb’s Approach

I won’t bother duplicating what Robb explains quite well, but this wasn’t a 10-minute solution for me, probably because I’m really still learning Javascript and Eleventy, and also because I’m using a Ghost back-end with packages managed via Yarn rather than npm.

So, in the few sections that follow I’ll try to briefly described what’s different about my approach versus Robb’s, along with what didn’t, and what ultimately did work.

`yarn`, Not `npm`

The first significant difference I encountered was the requirement that this blog project uses yarn, not npm for package management. Eventually I found that I just needed to replace Robb’s npm install pagefind command with a yarn add pagefind equivalent. I don’t really understand the details of that difference, but it worked.

eleventy.after Never Fires?

The biggest hurdle for me was trying to get this bit of Robb’s code to work.

const { execSync } = require('child_process')

module.exports = function(eleventyConfig) {
  eleventyConfig.on('eleventy.after', () => {
    execSync(`npx pagefind --source _site --glob \"**/*.html\"`, { encoding: 'utf-8' })
  })
}

In my case the --source parameter needed to be dist since this blog is generated into a directory by that name, and my configuration object is simply config rather than eleventyConfig. I made those simple changes and added my snippet to the .eleventy.js file, but found that it never triggered. I even replaced the execSync function call with a simple console.log statement and still got nothing. And still to this day, I have no idea why that didn’t work.

That snippet of code is supposed to be responsible for indexing all of the .html files after Eleventy has generated them. Well, I found out that everything else was working properly, but the _pagefind bundle generated by that indexing operation was just “missing”, since the index never was created. That made me wonder what would happen if I “forced” the system to index my content in a different manner…and that wonder ultimately worked.

Using `npx pagefind...` to Index Content

Having studied some of the Pagefind documentation I wondered what adding some post-processing to my build workflow might do, something like npx pagefind --source dist? That ultimately worked, but not in that original form.

I found that when I ran yarn start to locally rebuild the site, there was no generated ./dist/_pagefind bundle present, so I got no “Search” user interface elements, but there was an empty <div> where that belonged. So, with my localhost instance still running I opened a new terminal and ran npx pagefind --source dist in the project directory. Voila! The ./dist/_pagefind bundle magically appeared and the “Search” control was instantly visible in my localhost window! It was working, and it was beautiful.

Added a `pagefind` Script… Didn’t Work

Ok, with the above breakthrough in-hand I made this addition to the scripts: key in my package.json file:

  "pagefind": "npx pagefind --source dist"

Then I added the following to the end of my steps: key in .github/workflows/build-and-deploy.yml:

  - run: yarn pagefind

This worked to some extent, but wasn’t reliable and absolutely didn’t work when deployed to my DigitalOcean app. The inner-workings of yarn and build caching in these environments is still a DEEP mystery to me!

Extending My `encrypt` Script

What does work, both locally and on DigitalOcean, is essentially an “extension” of my old encrypt script. My old build workflow included these two elements of package.json and build-and-deploy.yml:

    "encrypt": "staticrypt ./dist/rebuild.html '***hidden***' --short -o ./dist/rebuild.html -f ./src/auth/login.html -t 'Summitt Dweller Blog - Rebuild' -i 'Please enter the passphrase.<br/>Hint: gh0st' --label-error 'Sorry, Ella says no. Try again.'"

…and…

  - run: yarn encrypt

I took note of the relative path spec for ./dist/rebuild.html in the encrypt command, reasoned that I should run pagefind in a similar fashion – but just before encryption – and came up with this change in package.json:

    "encrypt": "npx pagefind --source ./dist && staticrypt ./dist/rebuild.html '***hidden***' --short -o ./dist/rebuild.html -f ./src/auth/login.html -t 'Summitt Dweller Blog - Rebuild' -i 'Please enter the passphrase.<br/>Hint: gh0st' --label-error 'Sorry, Ella says no. Try again.'"

It works!

When I’m running locally my build-and-deploy.yml doesn’t apply so in that case I typically will do this IF I want my search index (and encryption) updated:

yarn start
yarn encrypt

When I rebuild in production on DigitalOcean my build-and-deploy.yml configuration IS run so including npx pagefind... as part of the yarn encrypt step does the trick.

Time to Test…

I’m going to save this post now and rebuild this blog on DigitalOcean, where with any luck, a search for a term like “DEEP” should return this post, and maybe one or two others. Drum roll please… Huzzah!

Man, Do I Feel Stupid

Previously I mentioned that my additon of a package.json script named pagefind didn’t work, and that my DigitalOcean builds kept trying to do yarn encrypt after yarn build. Well, turns out that’s because build-and-deploy.yml, where I had inserted the yarn pagefind step IS NOT in control of building my DigitalOcean app. In fact, it’s a .github workflow so I’m really not sure what role that file plays, if any?

I went looking in my DigitalOcean app configuration settings for yarn encrypt and found it, in the app’s “App Settings” subsection titled “App Spec”. When I view that “App Spec” I see this:

alerts:
- rule: DEPLOYMENT_FAILED
- rule: DOMAIN_FAILED
domains:
- domain: blog.summittdweller.com
  type: PRIMARY
  zone: summittdweller.com
envs:
- key: TZ
  scope: RUN_AND_BUILD_TIME
  value: America/Chicago
name: blog-summittdweller-11ty-ghost
region: nyc
static_sites:
- build_command: |-
    yarn build
    yarn encrypt
  environment_slug: node-js
  github:
    branch: main
    deploy_on_push: true
    repo: SummittDweller/blog-eleventy-ghost
  name: blog-eleventy-ghost
  output_dir: dist
  routes:
  - path: /
  source_dir: /

Note the two build_command: elements. Let’s change those two steps to three, like so:

- build_command: |-
    yarn build
    yarn pagefind
    yarn encrypt

Note that I captured the “App Spec” in a file named blog-summittdweller-11ty-ghost.yaml and I’ve saved that file in the project directory (where I hope it will do no harm). Attention: After making the aforementioned addition I almost forgot to upload this file back to DigitalOcean to replace our existing App Spec! Don’t be stupid like me!

Next, I needed a corresponding change in package.json, specifically these two lines (like I had them before, duh):

  "pagefind": "npx pagefind --source ./dist",
  "encrypt": "staticrypt ./dist/rebuild.html '***hidden***' --short -o ./dist/rebuild.html -f ./src/auth/login.html -t 'Summitt Dweller Blog - Rebuild' -i 'Please enter the passphrase.<br/>Hint: gh0st' --label-error 'Sorry, Ella says no. Try again.'"

Push to Rebuild

Ok, having modified and UPLOADEDblog-summittdweller-11ty-ghost.yaml, and having modified package.json, it came time to commit and push all my changes to see what happens. Another drum roll, please…