@Summitt Dweller’s Personal Blog - Development

Rebranding CV to HAT with Pagefind

Mark McFate — Sat, 25 Jan 2025 08:43:21 -0600

Today I’m rebranding the Tama-Toledo Community Visioning (CV) site to Hometown Action Team (HAT) and adding Pagefind.

When doing so I get this build error…

Warning: Running in pre-1.0 compatibility mode. Pagefind 1.0 changes the default output directory from /_pagefind/ to /pagefind/ but references to the /_pagefind/ URL were found on your site, and the output directory is unconfigured. To preserve your setup, the search files have been written twice, to both /_pagefind/ and /pagefind/

To remove this warning, either update your script and style references to the new /pagefind/ URL or run Pagefind with --output-subdir _pagefind to ensure pre-1.0 behaviour

So, I am going to try changing all _pagefind references to just ‘pagefind’. Wish me luck

A Newer Pagefind Enabled Blog!

Mark McFate — Sun, 22 Dec 2024 10:43:32 -0600

It’s deja vu all over again. My timeline module doesn’t work with the current rendition of https://blog.SummittDweller.com and I can’t figure out what’s broken. So, I’ve resurrected my old personal blog project and updated with the meager content it was missing. That project has a working timeline but no Pagefind search, yet.

So, how might I ONCE AGAIN approach combining Hugo with Pagefind in the cloud? Well, A Powerful Blog Setup with Hugo and NPM* by Tom Hombergs looked like a promising place to start. The process that Tom advocates leverages a neat little package called hugo-bin.

*The links provided above and below are to a Wayback Machine capture of the original post.

Once Again…That Was Too Easy!

Wrapping this Hugo blog site in an NPM package was so easy that I forgot to document it… and maybe I really didn’t need too. Time to catch up, so here’s a brief sysnopsis of what I did…

Following Tom’s Excellent Advice

I started by studying A Powerful Blog Setup with Hugo and NPM and quickly found that I could easily follow it almost verbatim. Since I already had npm and Hugo installed I skipped ahead to the section titled “Setting Up a Hugo Project” and then to the npx and git commands there. My experience, mostly in commands, looked like this on my Mac Mini workstation.

  cd ~/GitHub/
  # Named the new project npm-blog-SummittDweller so it would not confilct with any of my existing `blog...` directories
  mkdir npm-blog-SummittDweller
  cd npm-blog-SummittDweller
  npx hugo new site . --force
  # Copied the contents of my old project into the new one and then removed whatever I no longer needed
  cp -fr ~/GitHub/blogs-SummittDweller/. .  
  # Installed Pagefind...
  npm install pagefind
  git init
  # I created a new empty repo in GitHub called npm-blog-SummittDweller, then...
  git remote add origin https://github.com/SummittDweller/npm-blog-sd 
  git add . 
  git commit -m "Initial commit of new npm-wrapped Hugo project"
  git push

Having secured a new repo I moved to running Hugo locally, then polishing my package.json scripts as Tom did. I now have a package.json that reads like this:

{
  "name": "npm-blog-sd",
  "version": "1.0.0",
  "description": "My personal (SummittDweller) blog built with Hugo inside an NPM package.  December 2024.",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "build": "npm run hugo:build && npx pagefind --site public",
    "do:build": "hugo -d public && npx pagefind --site public",
    "azure:build": "hugo -d public --baseURL=\"https://azure-static-web-apps-ashy-rock-0f18efc0f\" && npx pagefind --site public",
    "clean": "npm run hugo:clean",
    "serve": "npm run hugo:build && npx pagefind --site public --output-subdir ../static/_pagefind && npm run hugo:serve",
    "hugo:build": "hugo -d public",
    "hugo:serve": "hugo server",
    "hugo:clean": "rm -rf build resources public"
  },
  "author": "Mark A. McFate",
  "license": "ISC",
  "dependencies": {
    "hugo-bin": "^0.137.0",
    "pagefind": "^1.2.0"
  }
}

It Works!

Now I just use npm run serve to build the Hugo site, index it with Pagefind, and serve it locally at http://localhost:1313.

A Staging Site in Azure

I can simply push changes to the main branch of the npm-blog-sd repo to trigger an Azure (aka GitHub Actions) rebuild, indexing and deployment of the site to this address.

Errors?

So, it took lots of trial-and-error to get the Azure configuration straight. First there were issues with deployment tokens, so I rewound the project and got that fixed. Then, since this blog uses SASS I needed to run the extended version of Hugo, or more specifically, hugo-bin-extended. I fixed that, apparently, with the addition of…

  "dependencies": {
    "hugo-bin-extended": "latest",
    "pagefind": "^1.2.0"
  }

…in package.json. That hugo-bin-extended key replaced hugo-bin and I set the version to latest.

Last of the Errors?

The changes mentioned above left me with two deprecation errors…

ERROR deprecated: site config key paginate was deprecated in Hugo v0.128.0 and will be removed in Hugo 0.141.0. Use pagination.pagerSize instead.
WARN  Raw HTML omitted while rendering "/github/workspace/content/about.md"; see https://gohugo.io/getting-started/configuration-markup/#rendererunsafe
You can suppress this warning by adding the following to your site configuration:
ignoreLogs = ['warning-goldmark-raw-html']
ERROR deprecated: .Site.Social was deprecated in Hugo v0.124.0 and will be removed in Hugo 0.141.0. Implement taxonomy 'social' or use .Site.Params.Social instead.

This post was very helpful for working with my submodule theme!

Success!

Well, somewhat. My site at https://ashy-rock-0f18efc0f.4.azurestaticapps.net/ is visible now, but the theme isn’t right. I suspect that is because my hugo.yml file still sets the baseURL to “https://blog.summittdweller.com” and we are not quite there yet. Setting the baseURL to “https://ashy-rock-0f18efc0f.4.azurestaticapps.net/" and let’s see what happens… Bingo! Now, I just need to add a search bar to expose the Pagefind results. Done!

What follows is obsolete, it's from `npm-rootstalk`, and does NOT apply to this blog!

A New Production Branch

Pushing to production is just as simple, I just have to push changes to the new production branch of the code and my app spec at DigitalOcean (see below) takes care of building, indexing, and deploying to https://rootstalk.grinnell.edu.

That DO app spec reads like this:

alerts:
- rule: DEPLOYMENT_FAILED
- rule: DOMAIN_FAILED
domains:
- domain: rootstalk.grinnell.edu
  type: PRIMARY
- domain: prairiejournal.grinnell.edu
  type: ALIAS
envs:
- key: TZ
  scope: RUN_AND_BUILD_TIME
  value: America/Chicago
- key: HUGO_MATOMO_ID
  scope: RUN_AND_BUILD_TIME
  value: "15"
ingress:
  rules:
  - component:
      name: npm-rootstalk
    match:
      path:
        prefix: /
name: npm-rootstalk
region: nyc
static_sites:
- build_command: npm run build
  environment_slug: node-js
  github:
    branch: production
    deploy_on_push: true
    repo: Digital-Grinnell/npm-rootstalk
  name: npm-rootstalk
  source_dir: /

It’s heavenly! And that’s all for now.

A New Pagefind Enabled Blog!

Mark McFate — Wed, 18 Dec 2024 08:31:49 -0600

So, node and npm are maybe not “all the rage” these days, but they do the job nicely for me.

*The links provided above and below are to a Wayback Machine capture of the original post.

That Was Too Easy!

Wrapping this Hugo blog site in an NPM package was so easy that I forgot to document it… and maybe I really didn’t need too. Time to catch up, so here’s a brief sysnopsis of what I did…

Following Tom’s Excellent Advice

cd ~/GitHub/
# Named the new project npm-blog-sd so it would not confilct with any of my existing `blog...` directories
mkdir npm-blog-sd
cd npm-blog-sd
npx hugo new site . --force
# Copied the contents of my old project into the new one and then removed whatever I no longer needed
cp -fr ~/GitHub/blog-sd/. .  
# Installed Pagefind...
npm install pagefind
git init
# I created a new empty repo in GitHub called npm-blog-sd, then...
git remote add origin https://github.com/SummittDweller/npm-blog-sd 
git add . 
git commit -m "Initial commit of new npm-wrapped Hugo project"
git push

Having secured a new repo I moved to running Hugo locally, then polishing my package.json scripts as Tom did. I now have a package.json that reads like this:

{
  "name": "npm-blog-sd",
  "version": "1.0.0",
  "description": "My personal (SummittDweller) blog built with Hugo inside an NPM package.  December 2024.",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "build": "npm run hugo:build && npx pagefind --site public",
    "do:build": "hugo -d public && npx pagefind --site public",
    "azure:build": "hugo -d public --baseURL=\"https://azure-static-web-apps-ashy-rock-0f18efc0f\" && npx pagefind --site public",
    "clean": "npm run hugo:clean",
    "serve": "npm run hugo:build && npx pagefind --site public --output-subdir ../static/_pagefind && npm run hugo:serve",
    "hugo:build": "hugo -d public",
    "hugo:serve": "hugo server",
    "hugo:clean": "rm -rf build resources public"
  },
  "author": "Mark A. McFate",
  "license": "ISC",
  "dependencies": {
    "hugo-bin": "^0.137.0",
    "pagefind": "^1.2.0"
  }
}

It Works!

Now I just use npm run serve to build the Hugo site, index it with Pagefind, and serve it locally at http://localhost:1313.

A Staging Site in Azure

I can simply push changes to the main branch of the npm-blog-sd repo to trigger an Azure (aka GitHub Actions) rebuild, indexing and deployment of the site to this address.

Errors?

  "dependencies": {
    "hugo-bin-extended": "latest",
    "pagefind": "^1.2.0"
  }

…in package.json. That hugo-bin-extended key replaced hugo-bin and I set the version to latest.

Last of the Errors?

The changes mentioned above left me with two deprecation errors…

ERROR deprecated: site config key paginate was deprecated in Hugo v0.128.0 and will be removed in Hugo 0.141.0. Use pagination.pagerSize instead.
WARN  Raw HTML omitted while rendering "/github/workspace/content/about.md"; see https://gohugo.io/getting-started/configuration-markup/#rendererunsafe
You can suppress this warning by adding the following to your site configuration:
ignoreLogs = ['warning-goldmark-raw-html']
ERROR deprecated: .Site.Social was deprecated in Hugo v0.124.0 and will be removed in Hugo 0.141.0. Implement taxonomy 'social' or use .Site.Params.Social instead.

This post was very helpful for working with my submodule theme!

Success!

What follows is obsolete, it's from `npm-rootstalk`, and does NOT apply to this blog!

A New Production Branch

That DO app spec reads like this:

alerts:
- rule: DEPLOYMENT_FAILED
- rule: DOMAIN_FAILED
domains:
- domain: rootstalk.grinnell.edu
  type: PRIMARY
- domain: prairiejournal.grinnell.edu
  type: ALIAS
envs:
- key: TZ
  scope: RUN_AND_BUILD_TIME
  value: America/Chicago
- key: HUGO_MATOMO_ID
  scope: RUN_AND_BUILD_TIME
  value: "15"
ingress:
  rules:
  - component:
      name: npm-rootstalk
    match:
      path:
        prefix: /
name: npm-rootstalk
region: nyc
static_sites:
- build_command: npm run build
  environment_slug: node-js
  github:
    branch: production
    deploy_on_push: true
    repo: Digital-Grinnell/npm-rootstalk
  name: npm-rootstalk
  source_dir: /

It’s heavenly! And that’s all for now.

Spinning Up StaticCMS

Mark McFate — Sat, 13 Jan 2024 15:40:23 +0000

Today is Saturday, January 13, 2024, and I won’t be able to watch the KC Chief’s wildcard playoff game vs. the Miami Dolphins tonight because it is streaming ONLY on Peacock, and I refuse to pay them for the “privilege” to watch it live. So, I’m going to do something productive instead. Let’s see if I can spin up an instance of StaticCMS to provide an OPEN CMS for the content editor(s) behind the Wieting Theatre website.

The Plan

So, the plan here involves establishing a “local-only” CMS for the Wieting Theatre website and running it on the new Ubuntu server in the Wieting’s projection booth. Editors should will be given login credentials to that server where they can run a local copy of the website (it’s a Hugo static site) AND a local copy of the CMS editor, make changes, preview them, and commit the changes to the project’s GitHub repo. The website, hosted with AWS Amplify is then automatically updated based on the updated content.

Along the Way

It’s worth noting that while looking at StaticCMS I also stumbled upon tabcms - A CMS that runs in your browser tab which might be of interest if “The Plan” doesn’t work.

First Steps - Start with a Template

I choose the guidance at Start with a Template to get started. My aim is to clone that repo and build it locally “as-is”, then see if I can drop my existing wieting-ss repo into the site directory and continue to make it work locally.

If that works, I’ll repeat the same on the Wieting’s Ubuntu server and keep pushing forward. Wish me luck?

Getting Started

Following the Local Development guidance found me cloning this template then…

╭─mark@Marks-Mac-Mini ~/GitHub/static-cms-hugo-netlify-template ‹main› 
╰─$ npm install 
npm WARN deprecated querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.

added 1567 packages, and audited 1568 packages in 2m

426 packages are looking for funding
  run `npm fund` for details

2 moderate severity vulnerabilities

To address all issues, run:
  npm audit fix

Run `npm audit` for details.
╭─mark@Marks-Mac-Mini ~/GitHub/static-cms-hugo-netlify-template ‹main*› 
╰─$ npm start

> static-cms-hugo-netlify-template@1.0.0 start
> run-p start:**


> static-cms-hugo-netlify-template@1.0.0 start:webpack
> webpack-dev-server --config webpack.dev.js --hot --stats-error-details


> static-cms-hugo-netlify-template@1.0.0 start:hugo
> hugo -d ../dist -s site -vw

WARN  --verbose was deprecated in Hugo v0.114.0 and will be removed in a future release. use --logLevel info
Start building sites … 
hugo v0.120.1-16fb2cae88eb6add7d12e9fbfcf01d8670e60a35 darwin/amd64 BuildDate=2023-10-30T16:44:31Z VendorInfo=gohugoio

INFO  copy static: syncing static files to /
INFO  build: running step process duration 962.376µs
INFO  build: running step assemble duration 4.712758ms
INFO  build: running step render duration 16.982673ms
INFO  build: running step postProcess duration 8.3µs

                   | EN  
-------------------+-----
  Pages            | 10  
  Paginator pages  |  0  
  Non-page files   |  0  
  Static files     | 43  
  Processed images |  0  
  Aliases          |  1  
  Sitemaps         |  1  
  Cleaned          |  0  

Built in 68 ms
Watching for changes in /Users/mark/GitHub/static-cms-hugo-netlify-template/site/{content,data,layouts,static}
Watching for config changes in /Users/mark/GitHub/static-cms-hugo-netlify-template/site/config.toml
Press Ctrl+C to stop
<i> [webpack-dev-server] Project is running at:
<i> [webpack-dev-server] Loopback: http://localhost:3000/
<i> [webpack-dev-server] On Your Network (IPv4): http://192.168.0.183:3000/
<i> [webpack-dev-server] On Your Network (IPv6): http://[fe80::1]:3000/
<i> [webpack-dev-server] Content not from webpack is served from '/Users/mark/GitHub/static-cms-hugo-netlify-template/dist' directory
<i> [webpack-dev-server] 404s will fallback to '/index.html'
<i> [webpack-dev-middleware] wait until bundle finished: /
assets by path *.js 18.3 MiB
  asset cms.js 18 MiB [emitted] (name: cms)
  asset main.js 300 KiB [emitted] (name: main)
asset main.css 113 KiB [emitted] (name: main)
asset admin/index.html 521 bytes [emitted]
asset fonts/.keep 0 bytes [emitted] [from: src/fonts/.keep] [copied]
Entrypoint main 413 KiB = main.css 113 KiB main.js 300 KiB
Entrypoint cms 18 MiB = cms.js
runtime modules 72.3 KiB 38 modules
modules by path ./node_modules/ 16.9 MiB 82 modules
modules by path ./src/ 133 KiB (javascript) 113 KiB (css/mini-extract)
  modules by path ./src/js/ 15.2 KiB 9 modules
  modules by path ./src/css/*.scss 118 KiB (javascript) 113 KiB (css/mini-extract)
    ./src/css/main.scss 2.78 KiB [built] [code generated]
    ./node_modules/to-string-loader/src/to-string.js!./node_modules/css-loader/dist/cjs.js!./node_modules/postcss-loader/dist/cjs.js!./node_modules/sass-loader/dist/cjs.js!./src/css/main.scss 429 bytes [built] [code generated]
    ./node_modules/mini-css-extract-plugin/dist/loader.js??ruleSet[1].rules[2].use[1]!./node_modules/css-loader/dist/cjs.js!./node_modules/postcss-loader/dist/cjs.js!./node_modules/sass-loader/dist/cjs.js!./src/css/main.scss 332 bytes [built] [code generated]
    + 2 modules
  ./src/index.js 124 bytes [built] [code generated]
asset modules 5.46 KiB
  data:image/png;base64,iVBORw0KGgoAAAAN.. 1.28 KiB [built] [code generated] [build time executed]
  data:image/png;base64,iVBORw0KGgoAAAAN.. 1.33 KiB [built] [code generated] [build time executed]
  data:image/jpeg;base64,/9j/4QAYRXhpZgAA.. 1.55 KiB [built] [code generated] [build time executed]
  data:image/png;base64,iVBORw0KGgoAAAAN.. 1.3 KiB [built] [code generated] [build time executed]
webpack 5.89.0 compiled successfully in 6191 ms
INFO  Received System Events: [CREATE        "/Users/mark/GitHub/static-cms-hugo-netlify-template/site/data/webpack.json" WRITE         "/Users/mark/GitHub/static-cms-hugo-netlify-template/site/data/webpack.json"]

Change detected, rebuilding site.
2024-01-13 10:55:38.259 -0600
Data changed WRITE         "/Users/mark/GitHub/static-cms-hugo-netlify-template/site/data/webpack.json"
Data changed WRITE         "/Users/mark/GitHub/static-cms-hugo-netlify-template/site/data/webpack.json"
INFO  build: running step process duration 114.033µs
INFO  build: running step assemble duration 277ns
INFO  build: running step render duration 6.244639ms
INFO  build: running step postProcess duration 1.831µs
Total in 6 ms

The demo site is indeed working at http://localhost:3000/ and a vist to http://localhost:3000/admin does indeed open a StaticCMS login that’s prompting me with the following image.

Let’s Go Local!

Next, I’m going to try and switch what we have above to “go local” using this guidance provided in the StaticCMS documenation. Fingers crossed for good luck…

Eureka! It Works!

So, I made the local_backend config changes as prescribed in site/static/admin/config.yml and that file now looks like this:

local_backend: 
  # when using a custom proxy server port
  url: http://localhost:8082/api/v1
  # when accessing the local site from a host other than 'localhost' or '127.0.0.1'
  allowed_hosts: ['192.168.0.1']

I also added a new .env file in the project root directory with a single line reading PORT=8082, matching the port number specified in the url: value from config.yml.

Now, with npx @staticcms/proxy-server running in a local terminal I’m able to run npm start (in my VSCode terminal) to launch the site at localhost:3000. Having done that I’m able to visit http://localhost:3000/admin where I get a new StaticCMS splash screen with a simple Login button, not a Netlify button. Yay!

Even Better News?

StaticCMS was built from NetlifyCMS so the site/static/admin/config.yml. I think I’ll be able to use the collections: portion of my old admin/config.yml file from wieting-one-click-hugo-cms to get a headstart on setting that up!

Next step… let’s see if I can drop the site directory from wieting-ss into this local project and still have a functioning local site. Again, fingers crossed for luck.

Fresh Restart

In order to start fresh I went back to the StaticJsCMS/static-cms-hugo-netlify-template repo and created a new fork (not a clone) in my Summitt-Services team space at https://github.com/Summitt-Services/static-cms-hugo-local-template. In the new fork I changed the -netlify- portion of the name to -local- as this new project is intended to have no Netlify association, only a “local” CMS will be supported.

Next, I cloned the new fork to my workstation and got it working locally again using the same steps I had taken above.

Copying `wieting-ss` to `static-cms-hugo-local-template`

For safe-keeping I created a new branch named wieting in ~/GitHub/static-cms-hugo-local-template where I did this:

╭─mark@Marks-Mac-Mini ~/GitHub/static-cms-hugo-local-template ‹wieting*› 
╰─$ rsync -aruvi ../wieting-ss/. site/.
...
>f++++++++++ themes/vanilla-bootstrap-hugo-theme/static/js/checkTweetDate.js
>f++++++++++ themes/vanilla-bootstrap-hugo-theme/static/js/feather.min.js
>f++++++++++ themes/vanilla-bootstrap-hugo-theme/static/js/jquery-3.3.1.slim.min.js

sent 282,877,469 bytes  received 470,944 bytes  9,605,030.95 bytes/sec
total size is 470,527,158  speedup is 1.66

Will it work? Let’s see… NOPE. It does produce a site but the old “Kaldi” content is still all I see. I even changed the backend.branch: value from main to wieting, but that did nothing. So, I’ve captured the output here so that I can have a look at what’s happening.

╭─mark@Marks-Mac-Mini ~/GitHub/static-cms-hugo-local-template ‹wieting*› 
╰─$ npm start                               

> static-cms-hugo-netlify-template@1.0.0 start
> run-p start:**


> static-cms-hugo-netlify-template@1.0.0 start:webpack
> webpack-dev-server --config webpack.dev.js --hot --stats-error-details


> static-cms-hugo-netlify-template@1.0.0 start:hugo
> hugo -d ../dist -s site -vw

WARN  --verbose was deprecated in Hugo v0.114.0 and will be removed in a future release. use --logLevel info
Start building sites … 
hugo v0.120.1-16fb2cae88eb6add7d12e9fbfcf01d8670e60a35 darwin/amd64 BuildDate=2023-10-30T16:44:31Z VendorInfo=gohugoio

INFO  copy static: syncing static files to /
INFO  build: running step process duration 4.236952ms
INFO  build: running step assemble duration 13.684119ms
WARN  found no layout file for "json" for kind "section": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
INFO  build: running step render duration 58.753135ms
INFO  build: running step postProcess duration 8.744µs

                   | EN   
-------------------+------
  Pages            |  66  
  Paginator pages  |   7  
  Non-page files   |   0  
  Static files     | 541  
  Processed images |   0  
  Aliases          |   7  
  Sitemaps         |   1  
  Cleaned          |   0  

Built in 579 ms
Watching for changes in /Users/mark/GitHub/static-cms-hugo-local-template/site/{archetypes,content,data,layouts,package.json,static,themes}
Watching for config changes in /Users/mark/GitHub/static-cms-hugo-local-template/site/config.toml, /Users/mark/GitHub/static-cms-hugo-local-template/site/config/_default
Press Ctrl+C to stop
<i> [webpack-dev-server] Project is running at:
<i> [webpack-dev-server] Loopback: http://localhost:3000/
<i> [webpack-dev-server] On Your Network (IPv4): http://192.168.0.183:3000/
<i> [webpack-dev-server] On Your Network (IPv6): http://[fe80::1]:3000/
<i> [webpack-dev-server] Content not from webpack is served from '/Users/mark/GitHub/static-cms-hugo-local-template/dist' directory
<i> [webpack-dev-server] 404s will fallback to '/index.html'
<i> [webpack-dev-middleware] wait until bundle finished: /
assets by path *.js 18.3 MiB
  asset cms.js 18 MiB [emitted] (name: cms)
  asset main.js 300 KiB [emitted] (name: main)
asset main.css 113 KiB [emitted] (name: main)
asset admin/index.html 521 bytes [emitted]
asset fonts/.keep 0 bytes [emitted] [from: src/fonts/.keep] [copied]
Entrypoint main 413 KiB = main.css 113 KiB main.js 300 KiB
Entrypoint cms 18 MiB = cms.js
runtime modules 72.3 KiB 38 modules
modules by path ./node_modules/ 16.9 MiB 82 modules
modules by path ./src/ 133 KiB (javascript) 113 KiB (css/mini-extract)
  modules by path ./src/js/ 15.2 KiB 9 modules
  modules by path ./src/css/*.scss 118 KiB (javascript) 113 KiB (css/mini-extract)
    ./src/css/main.scss 2.78 KiB [built] [code generated]
    ./node_modules/to-string-loader/src/to-string.js!./node_modules/css-loader/dist/cjs.js!./node_modules/postcss-loader/dist/cjs.js!./node_modules/sass-loader/dist/cjs.js!./src/css/main.scss 429 bytes [built] [code generated]
    ./node_modules/mini-css-extract-plugin/dist/loader.js??ruleSet[1].rules[2].use[1]!./node_modules/css-loader/dist/cjs.js!./node_modules/postcss-loader/dist/cjs.js!./node_modules/sass-loader/dist/cjs.js!./src/css/main.scss 332 bytes [built] [code generated]
    + 2 modules
  ./src/index.js 124 bytes [built] [code generated]
asset modules 5.46 KiB
  data:image/png;base64,iVBORw0KGgoAAAAN.. 1.28 KiB [built] [code generated] [build time executed]
  data:image/png;base64,iVBORw0KGgoAAAAN.. 1.33 KiB [built] [code generated] [build time executed]
  data:image/jpeg;base64,/9j/4QAYRXhpZgAA.. 1.55 KiB [built] [code generated] [build time executed]
  data:image/png;base64,iVBORw0KGgoAAAAN.. 1.3 KiB [built] [code generated] [build time executed]
webpack 5.89.0 compiled successfully in 5563 ms
INFO  Received System Events: [CHMOD         "/Users/mark/GitHub/static-cms-hugo-local-template/site/data/webpack.json" WRITE         "/Users/mark/GitHub/static-cms-hugo-local-template/site/data/webpack.json"]

Change detected, rebuilding site.
2024-01-13 13:15:08.329 -0600
Data changed WRITE         "/Users/mark/GitHub/static-cms-hugo-local-template/site/data/webpack.json"
INFO  build: running step process duration 186.604µs
INFO  build: running step assemble duration 244ns
WARN  found no layout file for "json" for kind "section": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
INFO  build: running step render duration 31.514265ms
INFO  build: running step postProcess duration 7.661µs
Total in 31 ms
^C<i> [webpack-dev-server] Gracefully shutting down. To force exit, press ^C again. Please wait...

Corrections Needed

So, the first thing I see here is that the new site directory contains the template’s content/_index.md file which naturally becomes the site’s home page. It contains only “Kaldi” content. So I tried removing that _index.md file and that changed the home page “title”, but very little else.

I think returing to the main branch of this project and starting over again but using only the content of wieting-ss in the new site/ directory might be wise. I did it like this:

╭─mark@Marks-Mac-Mini ~/GitHub
╰─$ rm -fr static-cms-hugo-local-template
╭─mark@Marks-Mac-Mini ~/GitHub
╰─$ git clone https://github.com/Summitt-Services/static-cms-hugo-local-template.git
Cloning into 'static-cms-hugo-local-template'...
remote: Enumerating objects: 574, done.
remote: Counting objects: 100% (155/155), done.
remote: Compressing objects: 100% (50/50), done.
remote: Total 574 (delta 122), reused 125 (delta 102), pack-reused 419
Receiving objects: 100% (574/574), 22.14 MiB | 23.94 MiB/s, done.
Resolving deltas: 100% (248/248), done.
╭─mark@Marks-Mac-Mini ~/GitHub
╰─$ cd static-cms-hugo-local-template
╭─mark@Marks-Mac-Mini ~/GitHub/static-cms-hugo-local-template ‹main›
╰─$ cp -fr site/static/admin ~
╭─mark@Marks-Mac-Mini ~/GitHub/static-cms-hugo-local-template ‹main›
╰─$ rm -fr site
╭─mark@Marks-Mac-Mini ~/GitHub/static-cms-hugo-local-template ‹main*›
╰─$ mkdir site
╭─mark@Marks-Mac-Mini ~/GitHub/static-cms-hugo-local-template ‹main*›
╰─$ cp -fr ../wieting-ss/. site/.
╭─mark@Marks-Mac-Mini ~/GitHub/static-cms-hugo-local-template ‹main*›
╰─$ cp -f ~/admin/config.yml site/static/admin/.

Next I launched the proxy server like so:

╭─mark@Marks-Mac-Mini ~/GitHub/static-cms-hugo-local-template ‹main*›
╰─$ npx @staticcms/proxy-server
info: Static CMS File System Proxy Server configured with /Users/mark/GitHub/static-cms-hugo-local-template
info: Static CMS Proxy Server listening on port 8082

Then in my VSCode terminal this:

╭─mark@Marks-Mac-Mini ~/GitHub/static-cms-hugo-local-template ‹main*› 
╰─$ npm start

> static-cms-hugo-netlify-template@1.0.0 start
> run-p start:**


> static-cms-hugo-netlify-template@1.0.0 start:webpack
> webpack-dev-server --config webpack.dev.js --hot --stats-error-details


> static-cms-hugo-netlify-template@1.0.0 start:hugo
> hugo -d ../dist -s site -vw

WARN  --verbose was deprecated in Hugo v0.114.0 and will be removed in a future release. use --logLevel info
Start building sites … 
hugo v0.120.1-16fb2cae88eb6add7d12e9fbfcf01d8670e60a35 darwin/amd64 BuildDate=2023-10-30T16:44:31Z VendorInfo=gohugoio

INFO  copy static: syncing static files to /
INFO  build: running step process duration 3.193622ms
INFO  build: running step assemble duration 25.748629ms
WARN  found no layout file for "json" for kind "section": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
INFO  build: running step render duration 70.992512ms
INFO  build: running step postProcess duration 9.94µs
<i> [webpack-dev-server] Project is running at:
<i> [webpack-dev-server] Loopback: http://localhost:3000/
<i> [webpack-dev-server] On Your Network (IPv4): http://192.168.0.183:3000/
<i> [webpack-dev-server] On Your Network (IPv6): http://[fe80::1]:3000/
<i> [webpack-dev-server] Content not from webpack is served from '/Users/mark/GitHub/static-cms-hugo-local-template/dist' directory
<i> [webpack-dev-server] 404s will fallback to '/index.html'

                   | EN   
-------------------+------
  Pages            |  57  
  Paginator pages  |   0  
  Non-page files   |   0  
  Static files     | 499  
  Processed images |   0  
  Aliases          |   1  
  Sitemaps         |   1  
  Cleaned          |   0  

Built in 1034 ms
Watching for changes in /Users/mark/GitHub/static-cms-hugo-local-template/site/{archetypes,content,layouts,package.json,static,themes}
Watching for config changes in /Users/mark/GitHub/static-cms-hugo-local-template/site/config/_default
Press Ctrl+C to stop
<i> [webpack-dev-middleware] wait until bundle finished: /
assets by path *.js 18.3 MiB
  asset cms.js 18 MiB [emitted] (name: cms)
  asset main.js 300 KiB [emitted] (name: main)
asset main.css 113 KiB [emitted] (name: main)
asset admin/index.html 521 bytes [emitted]
asset fonts/.keep 0 bytes [emitted] [from: src/fonts/.keep] [copied]
Entrypoint main 413 KiB = main.css 113 KiB main.js 300 KiB
Entrypoint cms 18 MiB = cms.js
runtime modules 72.3 KiB 38 modules
modules by path ./node_modules/ 16.9 MiB 82 modules
modules by path ./src/ 133 KiB (javascript) 113 KiB (css/mini-extract)
  modules by path ./src/js/ 15.2 KiB 9 modules
  modules by path ./src/css/*.scss 118 KiB (javascript) 113 KiB (css/mini-extract)
    ./src/css/main.scss 2.78 KiB [built] [code generated]
    ./node_modules/to-string-loader/src/to-string.js!./node_modules/css-loader/dist/cjs.js!./node_modules/postcss-loader/dist/cjs.js!./node_modules/sass-loader/dist/cjs.js!./src/css/main.scss 429 bytes [built] [code generated]
    ./node_modules/mini-css-extract-plugin/dist/loader.js??ruleSet[1].rules[2].use[1]!./node_modules/css-loader/dist/cjs.js!./node_modules/postcss-loader/dist/cjs.js!./node_modules/sass-loader/dist/cjs.js!./src/css/main.scss 332 bytes [built] [code generated]
    + 2 modules
  ./src/index.js 124 bytes [built] [code generated]
asset modules 5.46 KiB
  data:image/png;base64,iVBORw0KGgoAAAAN.. 1.28 KiB [built] [code generated] [build time executed]
  data:image/png;base64,iVBORw0KGgoAAAAN.. 1.33 KiB [built] [code generated] [build time executed]
  data:image/jpeg;base64,/9j/4QAYRXhpZgAA.. 1.55 KiB [built] [code generated] [build time executed]
  data:image/png;base64,iVBORw0KGgoAAAAN.. 1.3 KiB [built] [code generated] [build time executed]
webpack 5.89.0 compiled successfully in 6771 ms

Again, it works. Only this time the Wieting website appears at http://localhost:3000 and http://localhost:3000/admin works too, but with the wrong configuration. Still, that’s a WIN!

Time To Save My Progress

I really don’t want to dump 4000+ new files, many of them specific to the Wieting website, into my pristine (and working) https://github.com/Summitt-Services/static-cms-hugo-local-template GitHub repository. So, I’m going to clean up my local project by removing that which is no longer needed, and then I’m going to remove the .git directory and create/populate a new wieting-staticCMS private repository. After creating that empty repository the process looked like this:

╭─mark@Marks-Mac-Mini ~/GitHub/static-cms-hugo-local-template ‹main*› 
╰─$ rm -fr site/.git 
╭─mark@Marks-Mac-Mini ~/GitHub/static-cms-hugo-local-template ‹main*› 
╰─$ rm -fr .git
╭─mark@Marks-Mac-Mini ~/GitHub/static-cms-hugo-local-template 
╰─$ git init
Initialized empty Git repository in /Users/mark/GitHub/static-cms-hugo-local-template/.git/
╭─mark@Marks-Mac-Mini ~/GitHub/static-cms-hugo-local-template 
╰─$ git add .
╭─mark@Marks-Mac-Mini ~/GitHub/static-cms-hugo-local-template 
╰─$ git commit -m "first commit"
╭─mark@Marks-Mac-Mini ~/GitHub/static-cms-hugo-local-template 
╰─$ git branch -M main
╭─mark@Marks-Mac-Mini ~/GitHub/static-cms-hugo-local-template 
╰─$ git remote add origin https://github.com/SummittDweller/wieting-staticCMS.git
╭─mark@Marks-Mac-Mini ~/GitHub/static-cms-hugo-local-template 
╰─$ git push -u origin main
...
Enumerating objects: 2979, done.
Counting objects: 100% (2979/2979), done.
Delta compression using up to 6 threads
Compressing objects: 100% (2855/2855), done.
Writing objects: 100% (2979/2979), 95.87 MiB | 2.65 MiB/s, done.
Total 2979 (delta 638), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (638/638), done.
To https://github.com/SummittDweller/wieting-staticCMS.git
 * [new branch]      main -> main
branch 'main' set up to track 'origin/main'.

wieting-staticCMS

With the new GitHub repo in place let’s take it for a local spin.

╭─mark@Marks-Mac-Mini ~/GitHub
╰─$ git clone https://github.com/SummittDweller/wieting-staticCMS.git
Cloning into 'wieting-staticCMS'...
remote: Enumerating objects: 2979, done.
remote: Total 2979 (delta 0), reused 0 (delta 0), pack-reused 2979
Receiving objects: 100% (2979/2979), 95.87 MiB | 26.77 MiB/s, done.
Resolving deltas: 100% (638/638), done.
Updating files: 100% (4834/4834), done.
╭─mark@Marks-Mac-Mini ~/GitHub
╰─$ cd wieting-staticCMS
╭─mark@Marks-Mac-Mini ~/GitHub/wieting-staticCMS ‹main›
╰─$ npm install
npm WARN deprecated querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.

added 1567 packages, and audited 1568 packages in 1m

426 packages are looking for funding
  run `npm fund` for details

2 moderate severity vulnerabilities

To address all issues, run:
  npm audit fix

Run `npm audit` for details.
╭─mark@Marks-Mac-Mini ~/GitHub/wieting-staticCMS ‹main›
╰─$ npx @staticcms/proxy-server
info: Static CMS File System Proxy Server configured with /Users/mark/GitHub/wieting-staticCMS
info: Static CMS Proxy Server listening on port 8082

Then, in a differnt terminal window…

╭─mark@Marks-Mac-Mini ~/GitHub/wieting-staticCMS ‹main›
╰─$ npm start

> static-cms-hugo-netlify-template@1.0.0 start
> run-p start:**


> static-cms-hugo-netlify-template@1.0.0 start:hugo
> hugo -d ../dist -s site -vw


> static-cms-hugo-netlify-template@1.0.0 start:webpack
> webpack-dev-server --config webpack.dev.js --hot --stats-error-details

WARN  --verbose was deprecated in Hugo v0.114.0 and will be removed in a future release. use --logLevel info
Start building sites …
hugo v0.120.1-16fb2cae88eb6add7d12e9fbfcf01d8670e60a35 darwin/amd64 BuildDate=2023-10-30T16:44:31Z VendorInfo=gohugoio

INFO  copy static: syncing static files to /
INFO  build: running step process duration 3.975463ms
INFO  build: running step assemble duration 19.211854ms
WARN  found no layout file for "json" for kind "section": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
INFO  build: running step render duration 54.091014ms
INFO  build: running step postProcess duration 8.597µs

                   | EN
-------------------+------
  Pages            |  57
  Paginator pages  |   0
  Non-page files   |   0
  Static files     | 499
  Processed images |   0
  Aliases          |   1
  Sitemaps         |   1
  Cleaned          |   0

Built in 717 ms
Watching for changes in /Users/mark/GitHub/wieting-staticCMS/site/{archetypes,content,layouts,package.json,static,themes}
Watching for config changes in /Users/mark/GitHub/wieting-staticCMS/site/config/_default
Press Ctrl+C to stop
<i> [webpack-dev-server] Project is running at:
<i> [webpack-dev-server] Loopback: http://localhost:3000/
<i> [webpack-dev-server] On Your Network (IPv4): http://192.168.0.183:3000/
<i> [webpack-dev-server] On Your Network (IPv6): http://[fe80::1]:3000/
<i> [webpack-dev-server] Content not from webpack is served from '/Users/mark/GitHub/wieting-staticCMS/dist' directory
<i> [webpack-dev-server] 404s will fallback to '/index.html'
<i> [webpack-dev-middleware] wait until bundle finished: /
assets by path *.js 18.3 MiB
  asset cms.js 18 MiB [emitted] (name: cms)
  asset main.js 300 KiB [emitted] (name: main)
asset main.css 113 KiB [emitted] (name: main)
asset admin/index.html 521 bytes [emitted]
asset fonts/.keep 0 bytes [emitted] [from: src/fonts/.keep] [copied]
Entrypoint main 413 KiB = main.css 113 KiB main.js 300 KiB
Entrypoint cms 18 MiB = cms.js
runtime modules 72.3 KiB 38 modules
modules by path ./node_modules/ 16.9 MiB 82 modules
modules by path ./src/ 133 KiB (javascript) 113 KiB (css/mini-extract)
  modules by path ./src/js/ 15.2 KiB 9 modules
  modules by path ./src/css/*.scss 118 KiB (javascript) 113 KiB (css/mini-extract)
    ./src/css/main.scss 2.78 KiB [built] [code generated]
    ./node_modules/to-string-loader/src/to-string.js!./node_modules/css-loader/dist/cjs.js!./node_modules/postcss-loader/dist/cjs.js!./node_modules/sass-loader/dist/cjs.js!./src/css/main.scss 429 bytes [built] [code generated]
    ./node_modules/mini-css-extract-plugin/dist/loader.js??ruleSet[1].rules[2].use[1]!./node_modules/css-loader/dist/cjs.js!./node_modules/postcss-loader/dist/cjs.js!./node_modules/sass-loader/dist/cjs.js!./src/css/main.scss 332 bytes [built] [code generated]
    + 2 modules
  ./src/index.js 124 bytes [built] [code generated]
asset modules 5.46 KiB
  data:image/png;base64,iVBORw0KGgoAAAAN.. 1.28 KiB [built] [code generated] [build time executed]
  data:image/png;base64,iVBORw0KGgoAAAAN.. 1.33 KiB [built] [code generated] [build time executed]
  data:image/jpeg;base64,/9j/4QAYRXhpZgAA.. 1.55 KiB [built] [code generated] [build time executed]
  data:image/png;base64,iVBORw0KGgoAAAAN.. 1.3 KiB [built] [code generated] [build time executed]
webpack 5.89.0 compiled successfully in 6051 ms

And again, IT WORKS!

Time to merge the old CMS config from site/static/.admin-netlify/config.yml into the new config at site/static/admin/config.yml. Then a few changes will be needed since the site’s content structure has changed somewhat from the old days.

Time for a New Static CMS Approach

Mark McFate — Sat, 23 Dec 2023 12:04:41 +0000

Wow, it’s been way too long since I posted here. So much has happened, and been missed along the way. Time to fix that and I’m going to start with a re-write of the Wieting Theatre website using staticcms/core. Watch for a full blog post coming soon!

Improving My Azure Blob Storage

Mark McFate — Mon, 20 Nov 2023 19:00:04 +0000

I use Azure Blob Storage quite a bit these days, at work and here at home. Recently a student I’ve been working with turned my attention to this post from Microsoft and this week I’m going to take a crack at putting it to use, perhaps in association with this blog.

Building My First Steamlit App

Mark McFate — Mon, 24 Jul 2023 18:44:51 +0000

I’ve been wanting to give Streamlit a try for some time now. Recently, I found Streamlit DataScience Apps, a GitHub repo full of Streamlit apps, written by Jcharis. In particular, I thought I would take the Job_Search_App_with_Streamlit_Forms_n_GithubAPI code for a spin.

First step, I created a new ~/GitHub/ code directory on my Mac; I named my directory ~/GitHub/job-search-app-with-streamlit-forms. The plan was to work in that directory inside VSCode with all the project requirements housed in a Python virtual environment (.venv) there.

The command stream in my terminal looked like this:

╭─mark@Marks-Mac-Mini ~
╰─$ cd ~/GitHub
╭─mark@Marks-Mac-Mini ~/GitHub
╰─$ mkdir job-search-app-with-streamlit-forms
╭─mark@Marks-Mac-Mini ~/GitHub
╰─$ cd job-search-app-with-streamlit-forms
╭─mark@Marks-Mac-Mini ~/GitHub/job-search-app-with-streamlit-forms
╰─$ code .

That last command opened the new directory in VSCode where I had all the tools I needed, including a new terminal window. In that window I continued with…

╭─mark@Marks-Mac-Mini ~/GitHub/job-search-app-with-streamlit-forms 
╰─$ python3 -m venv .venv

╭─mark@Marks-Mac-Mini ~/GitHub/job-search-app-with-streamlit-forms 
╰─$ source .venv/bin/activate

(.venv) ╭─mark@Marks-Mac-Mini ~/GitHub/job-search-app-with-streamlit-forms 
╰─$ pip3 install streamlit

Collecting streamlit
  Downloading streamlit-1.25.0-py2.py3-none-any.whl (8.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.1/8.1 MB 26.6 MB/s eta 0:00:00
Collecting altair<6,>=4.0
  Downloading altair-5.0.1-py3-none-any.whl (471 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 471.5/471.5 kB 13.9 MB/s eta 0:00:00
Collecting blinker<2,>=1.0.0
  Downloading blinker-1.6.2-py3-none-any.whl (13 kB)
Collecting cachetools<6,>=4.0
  Downloading cachetools-5.3.1-py3-none-any.whl (9.3 kB)
Collecting click<9,>=7.0
  Downloading click-8.1.6-py3-none-any.whl (97 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.9/97.9 kB 3.7 MB/s eta 0:00:00
Collecting importlib-metadata<7,>=1.4
  Downloading importlib_metadata-6.8.0-py3-none-any.whl (22 kB)
Collecting numpy<2,>=1.19.3
  Downloading numpy-1.25.1-cp311-cp311-macosx_10_9_x86_64.whl (20.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.0/20.0 MB 30.2 MB/s eta 0:00:00
Collecting packaging<24,>=16.8
  Downloading packaging-23.1-py3-none-any.whl (48 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.9/48.9 kB 1.6 MB/s eta 0:00:00
Collecting pandas<3,>=1.3.0
  Downloading pandas-2.0.3-cp311-cp311-macosx_10_9_x86_64.whl (11.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.6/11.6 MB 38.2 MB/s eta 0:00:00
Collecting pillow<10,>=7.1.0
  Downloading Pillow-9.5.0-cp311-cp311-macosx_10_10_x86_64.whl (3.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 23.5 MB/s eta 0:00:00
Collecting protobuf<5,>=3.20
  Downloading protobuf-4.23.4-cp37-abi3-macosx_10_9_universal2.whl (400 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 400.3/400.3 kB 12.9 MB/s eta 0:00:00
Collecting pyarrow>=6.0
  Downloading pyarrow-12.0.1-cp311-cp311-macosx_10_14_x86_64.whl (24.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 24.7/24.7 MB 19.0 MB/s eta 0:00:00
Collecting pympler<2,>=0.9
  Downloading Pympler-1.0.1-py3-none-any.whl (164 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 164.8/164.8 kB 5.9 MB/s eta 0:00:00
Collecting python-dateutil<3,>=2.7.3
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 7.9 MB/s eta 0:00:00
Collecting requests<3,>=2.18
  Downloading requests-2.31.0-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 2.3 MB/s eta 0:00:00
Collecting rich<14,>=10.14.0
  Downloading rich-13.4.2-py3-none-any.whl (239 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 239.4/239.4 kB 8.8 MB/s eta 0:00:00
Collecting tenacity<9,>=8.1.0
  Downloading tenacity-8.2.2-py3-none-any.whl (24 kB)
Collecting toml<2,>=0.10.1
  Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting typing-extensions<5,>=4.1.0
  Downloading typing_extensions-4.7.1-py3-none-any.whl (33 kB)
Collecting tzlocal<5,>=1.1
  Downloading tzlocal-4.3.1-py3-none-any.whl (20 kB)
Collecting validators<1,>=0.2
  Downloading validators-0.20.0.tar.gz (30 kB)
  Preparing metadata (setup.py) ... done
Collecting gitpython!=3.1.19,<4,>=3.0.7
  Downloading GitPython-3.1.32-py3-none-any.whl (188 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 188.5/188.5 kB 6.9 MB/s eta 0:00:00
Collecting pydeck<1,>=0.8
  Downloading pydeck-0.8.0-py2.py3-none-any.whl (4.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.7/4.7 MB 28.7 MB/s eta 0:00:00
Collecting tornado<7,>=6.0.3
  Downloading tornado-6.3.2-cp38-abi3-macosx_10_9_x86_64.whl (422 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 422.6/422.6 kB 12.7 MB/s eta 0:00:00
Collecting jinja2
  Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.1/133.1 kB 5.4 MB/s eta 0:00:00
Collecting jsonschema>=3.0
  Downloading jsonschema-4.18.4-py3-none-any.whl (80 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 81.0/81.0 kB 3.0 MB/s eta 0:00:00
Collecting toolz
  Downloading toolz-0.12.0-py3-none-any.whl (55 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.8/55.8 kB 2.1 MB/s eta 0:00:00
Collecting gitdb<5,>=4.0.1
  Downloading gitdb-4.0.10-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.7/62.7 kB 2.1 MB/s eta 0:00:00
Collecting zipp>=0.5
  Downloading zipp-3.16.2-py3-none-any.whl (7.2 kB)
Collecting pytz>=2020.1
  Using cached pytz-2023.3-py2.py3-none-any.whl (502 kB)
Collecting tzdata>=2022.1
  Downloading tzdata-2023.3-py2.py3-none-any.whl (341 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 341.8/341.8 kB 10.8 MB/s eta 0:00:00
Collecting six>=1.5
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting charset-normalizer<4,>=2
  Downloading charset_normalizer-3.2.0-cp311-cp311-macosx_10_9_x86_64.whl (125 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 125.1/125.1 kB 4.5 MB/s eta 0:00:00
Collecting idna<4,>=2.5
  Downloading idna-3.4-py3-none-any.whl (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 2.1 MB/s eta 0:00:00
Collecting urllib3<3,>=1.21.1
  Downloading urllib3-2.0.4-py3-none-any.whl (123 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 123.9/123.9 kB 4.5 MB/s eta 0:00:00
Collecting certifi>=2017.4.17
  Downloading certifi-2023.7.22-py3-none-any.whl (158 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 158.3/158.3 kB 5.7 MB/s eta 0:00:00
Collecting markdown-it-py>=2.2.0
  Downloading markdown_it_py-3.0.0-py3-none-any.whl (87 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 87.5/87.5 kB 3.3 MB/s eta 0:00:00
Collecting pygments<3.0.0,>=2.13.0
  Downloading Pygments-2.15.1-py3-none-any.whl (1.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 23.0 MB/s eta 0:00:00
Collecting pytz-deprecation-shim
  Downloading pytz_deprecation_shim-0.1.0.post0-py2.py3-none-any.whl (15 kB)
Collecting decorator>=3.4.0
  Downloading decorator-5.1.1-py3-none-any.whl (9.1 kB)
Collecting smmap<6,>=3.0.1
  Downloading smmap-5.0.0-py3-none-any.whl (24 kB)
Collecting MarkupSafe>=2.0
  Downloading MarkupSafe-2.1.3-cp311-cp311-macosx_10_9_x86_64.whl (13 kB)
Collecting attrs>=22.2.0
  Downloading attrs-23.1.0-py3-none-any.whl (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.2/61.2 kB 2.1 MB/s eta 0:00:00
Collecting jsonschema-specifications>=2023.03.6
  Downloading jsonschema_specifications-2023.7.1-py3-none-any.whl (17 kB)
Collecting referencing>=0.28.4
  Downloading referencing-0.30.0-py3-none-any.whl (25 kB)
Collecting rpds-py>=0.7.1
  Downloading rpds_py-0.9.2-cp311-cp311-macosx_10_7_x86_64.whl (311 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 311.9/311.9 kB 9.3 MB/s eta 0:00:00
Collecting mdurl~=0.1
  Downloading mdurl-0.1.2-py3-none-any.whl (10.0 kB)
Installing collected packages: pytz, zipp, urllib3, tzdata, typing-extensions, tornado, toolz, toml, tenacity, smmap, six, rpds-py, pympler, pygments, protobuf, pillow, packaging, numpy, mdurl, MarkupSafe, idna, decorator, click, charset-normalizer, certifi, cachetools, blinker, attrs, validators, requests, referencing, pytz-deprecation-shim, python-dateutil, pyarrow, markdown-it-py, jinja2, importlib-metadata, gitdb, tzlocal, rich, pydeck, pandas, jsonschema-specifications, gitpython, jsonschema, altair, streamlit
  DEPRECATION: validators is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for validators ... done
Successfully installed MarkupSafe-2.1.3 altair-5.0.1 attrs-23.1.0 blinker-1.6.2 cachetools-5.3.1 certifi-2023.7.22 charset-normalizer-3.2.0 click-8.1.6 decorator-5.1.1 gitdb-4.0.10 gitpython-3.1.32 idna-3.4 importlib-metadata-6.8.0 jinja2-3.1.2 jsonschema-4.18.4 jsonschema-specifications-2023.7.1 markdown-it-py-3.0.0 mdurl-0.1.2 numpy-1.25.1 packaging-23.1 pandas-2.0.3 pillow-9.5.0 protobuf-4.23.4 pyarrow-12.0.1 pydeck-0.8.0 pygments-2.15.1 pympler-1.0.1 python-dateutil-2.8.2 pytz-2023.3 pytz-deprecation-shim-0.1.0.post0 referencing-0.30.0 requests-2.31.0 rich-13.4.2 rpds-py-0.9.2 six-1.16.0 smmap-5.0.0 streamlit-1.25.0 tenacity-8.2.2 toml-0.10.2 toolz-0.12.0 tornado-6.3.2 typing-extensions-4.7.1 tzdata-2023.3 tzlocal-4.3.1 urllib3-2.0.4 validators-0.20.0 zipp-3.16.2

[notice] A new release of pip is available: 23.0.1 -> 23.2.1
[notice] To update, run: pip install --upgrade pip

(.venv) ╭─mark@Marks-Mac-Mini ~/GitHub/job-search-app-with-streamlit-forms 
╰─$ pip install --upgrade pip

Requirement already satisfied: pip in ./.venv/lib/python3.11/site-packages (23.0.1)
Collecting pip
  Downloading pip-23.2.1-py3-none-any.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 12.6 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 23.0.1
    Uninstalling pip-23.0.1:
      Successfully uninstalled pip-23.0.1
Successfully installed pip-23.2.1

(.venv) ╭─mark@Marks-Mac-Mini ~/GitHub/job-search-app-with-streamlit-forms 
╰─$ pip3 install requests

Requirement already satisfied: requests in ./.venv/lib/python3.11/site-packages (2.31.0)
Requirement already satisfied: charset-normalizer<4,>=2 in ./.venv/lib/python3.11/site-packages (from requests) (3.2.0)
Requirement already satisfied: idna<4,>=2.5 in ./.venv/lib/python3.11/site-packages (from requests) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in ./.venv/lib/python3.11/site-packages (from requests) (2.0.4)
Requirement already satisfied: certifi>=2017.4.17 in ./.venv/lib/python3.11/site-packages (from requests) (2023.7.22)

(.venv) ╭─mark@Marks-Mac-Mini ~/GitHub/job-search-app-with-streamlit-forms 
╰─$ streamlit app
Usage: streamlit [OPTIONS] COMMAND [ARGS]...
Try 'streamlit --help' for help.

Error: No such command 'app'.

(.venv) ╭─mark@Marks-Mac-Mini ~/GitHub/job-search-app-with-streamlit-forms 
╰─$ streamlit run app.py

  You can now view your Streamlit app in your browser.

  Local URL: http://localhost:8501
  Network URL: http://192.168.0.183:8501

  For better performance, install the Watchdog module:

  $ xcode-select --install
  $ pip install watchdog
            
2023-07-24 20:58:17.240 Uncaught app exception
Traceback (most recent call last):
  File "/Users/mark/GitHub/job-search-app-with-streamlit-forms/.venv/lib/python3.11/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 552, in _run_script
    exec(code, module.__dict__)
  File "/Users/mark/GitHub/job-search-app-with-streamlit-forms/app.py", line 116, in <module>
    main()
  File "/Users/mark/GitHub/job-search-app-with-streamlit-forms/app.py", line 43, in main
    nav1,nav2,nav3 = st.beta_columns([3,2,1])
                     ^^^^^^^^^^^^^^^
AttributeError: module 'streamlit' has no attribute 'beta_columns'

Mackenzie says I need to remove the beta_ in two places in the code… Done.

Launching the app again with streamlit run app.py after a moment or two it works!

Upgrading Matomo PHP

Mark McFate — Mon, 12 Jun 2023 19:30:18 +0000

This afternoon I’m adding https://digital.grinnell.edu (site ID is 17) to my own Matomo instance and I’m prompted to upgrade PHP on that Matomo droplet (DigitalOcean). I did so quite easily following Upgrade PHP from 7.2 to 7.4 on CentOS 8 and Apache. I’ve also updated my Drupal version, and some modules, in https://digital.grinnell.edu along with adding the matomo module, but that module is now reporting:

    The validation of "http://analytics.summittservices.com/matomo.php" failed with error "Error opening socket ssl://analytics.summittservices.com:443" (HTTP code 0).
    The validation of "https://analytics.summittservices.com/matomo.php" failed with error "Error opening socket ssl://analytics.summittservices.com:443" (HTTP code 0).

Hmmm…wonder what’s up with that?

So, I tried this…

Last login: Mon Jun 12 19:21:45 2023 from 162.243.190.66
[root@centos-s-1vcpu-1gb-nyc1-01 ~]# sudo certbot renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/summittservices.com-0001.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for summittservices.com and analytics.summittservices.com
Reloading nginx server after certificate renewal

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/summittservices.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for summittservices.com and www.summittservices.com
Reloading nginx server after certificate renewal

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations, all renewals succeeded: 
  /etc/letsencrypt/live/summittservices.com-0001/fullchain.pem (success)
  /etc/letsencrypt/live/summittservices.com/fullchain.pem (success)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

…and… NO more validation errors! Yay!

Streamlit Apps Fork

Mark McFate — Mon, 05 Jun 2023 12:31:46 +0000

This morning I found some new Streamlit resources from developer Jcharis, and I forked his Jcharis/Streamlit_DataScience_Apps to make my own copy at SummittDweller/Streamlit_DataScience_Apps

Now I’m going to try and get the component Simple_CRuD_Blog_App_with_Streamlit working locally as a proof-of-concept. Since I can’t “clone” the single project repo, and a “sparse checkout” seems wasteful since I’ll never push back to the original project, I’m going to git init an entirely new local named simple-CRuD-blog-app-with-streamlit and just copy/paste the contents.

It works!

Fixing Wieting Guild Pages

Mark McFate — Sat, 03 Jun 2023 23:20:26 +0000

Just the other day I got an updated copy of the Wieting Theatre’s volunteer roster and assignment schedule. As I’m trying to post them to the theatre’s management website (it’s password protected) I keep having issues with npm and dependencies that won’t compile. The fix, thus far has included a local command stream like this:

cd wieting-guild-pages
git pull
code .
npm run build
npm run start 
  --> failed
npm update
npm audit fix
npm run build
npm run start
  --> failed
npm audit fix --force
npm run build
  --> failed
npm install axios
npm run build
  --> failed
npm update
npm run build

The last errors had this in common:

[11ty] Unhandled rejection in promise: (more in DEBUG output)
[11ty] connect ETIMEDOUT 54.243.162.8:443 (via Error)
[11ty] 
[11ty] Original error stack trace: Error: connect ETIMEDOUT 54.243.162.8:443
[11ty]     at AxiosError.from (/Users/mark/GitHub/wieting-guild-pages/node_modules/axios/dist/node/axios.cjs:836:14)

One explanation I found:

The ETIMEDOUT error is thrown by the Node.js runtime when a connection or HTTP request is not closed properly after some time. You might encounter this error from time to time if you configured a timeout on your outgoing HTTP requests. The general solution to this issue is to catch the error and repeat the request, preferably using an exponential backoff strategy so that a waiting period is added between subsequent retries until the request eventually succeeds, or the maximum amount of retries is reached. If you encounter this error frequently, try to investigate your request timeout settings and choose a more appropriate value for the endpoint if possible.

I also found a GitHub issues/bug report related to Axios and this error at https://github.com/SillyTavern/SillyTavern/issues/432. So, maybe it’s not just me?

Streamlit Ghost Azure Resources

Mark McFate — Thu, 01 Jun 2023 13:49:18 +0000

I’ve got Mackenzie looking at Streamlit as a means of building a mobile-friendly app that I can use to create posts like this one. That app will need to allow me to edit a TITLE and Markdown BODY fields, capture location and time, and post all of that information to my personal blog… preferably from my cell phone. I’ve captured some possible resources and guidance in https://www.one-tab.com/page/71iaDADxT-eLEOx15qeVnQ.

Wrap EVERYTHING in NPM!

Mark McFate — Fri, 19 May 2023 22:27:00 +0000

So, node and npm seem to be all the rage these days, and perhaps for good reason. I recently fell in love with Eleventy/11ty over Hugo because it’s Javascript, not Go, and it’s elegantly simple with tons of flexibility. I recently tried to add Pagefind search to a Hugo static web site (see https://static.grinnell.edu/dlad-blog/posts/143-significant-rootstalk-retooling/). If Rootstalk, an Azure Static Web App was framed in node.js, as both Eleventy and Pagefind are, there would be no problem. The Azure scripts used to deploy those frameworks are far more customizable than Hugo, and there’s documentation to prove it.

So, how might I approach combining Hugo with Pagefind in the cloud? Well, A Powerful Blog Setup with Hugo and NPM* by Tom Hombergs looked like a promising place to start. The process that Tom advocates leverages a neat little package called hugo-bin.

*The links provided above and below are to a Wayback Machine capture of the original post.

That Was Too Easy!

Wrapping my Hugo site in an NPM package was so easy that I forgot to document it… and maybe I really didn’t need too. Time to catch up, so here’s a brief sysnopsis of what I did…

Following Tom’s Excellent Advice

cd ~/GitHub/
# Named the new project npm-rootstalk so it would not confilct with any of my existing `rootstalk...` directories
mkdir npm-rootstalk      
cd npm-rootstalk
npx hugo new site . --force
# Copied the contents of my old project into the new one and then removed whatever I no longer needed
cp -fr ~/GitHub/rootstalk/. .  
# Installed Pagefind...
npm install pagefind
git init
# I created a new empty repo in GitHub called npm-rootstalk, then...
git remote add origin https://github.com/Digital-Grinnell/npm-rootstalk 
git add . 
git commit -m "Initial commit of new npm-wrapped Hugo project"
git push

Having secured a new repo I moved to running Hugo locally, then polishing my package.json scripts as Tom did. I now have a package.json that reads like this:

{
  "name": "npm-rootstalk",
  "version": "1.0.0",
  "description": "Rootstalk website built with Hugo and the Lightbi theme inside an NPM package.  May 2023.",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "build": "npm run hugo:build && npx pagefind --source public",
    "do:build": "hugo -d public && npx pagefind --source public",
    "azure:build": "hugo -d public --baseURL=\"https://yellow-wave-0e513e510.3.azurestaticapps.net/\" && npx pagefind --source public",
    "clean": "npm run hugo:clean",
    "serve": "npm run hugo:build && npx pagefind --source public --bundle-dir ../static/_pagefind && npm run hugo:serve",
    "hugo:build": "hugo -d public",
    "hugo:serve": "hugo server",
    "hugo:clean": "rm -rf build resources public"
  },
  "author": "Mark A. McFate",
  "license": "ISC",
  "dependencies": {
    "hugo-bin": "^0.102.0",
    "pagefind": "^0.12.0"
  }
}

It Works!

Now I just use npm run serve to build the Hugo site, index it with Pagefind, and serve it locally at http://localhost:1313.

A Staging Site in Azure

I can simply push changes to the main branch of the npm-rootstalk repo to trigger an Azure (aka GitHub Actions) rebuild, indexing and deployment of the site to this address.

A New Production Branch

That DO app spec reads like this:

alerts:
- rule: DEPLOYMENT_FAILED
- rule: DOMAIN_FAILED
domains:
- domain: rootstalk.grinnell.edu
  type: PRIMARY
- domain: prairiejournal.grinnell.edu
  type: ALIAS
envs:
- key: TZ
  scope: RUN_AND_BUILD_TIME
  value: America/Chicago
- key: HUGO_MATOMO_ID
  scope: RUN_AND_BUILD_TIME
  value: "15"
ingress:
  rules:
  - component:
      name: npm-rootstalk
    match:
      path:
        prefix: /
name: npm-rootstalk
region: nyc
static_sites:
- build_command: npm run build
  environment_slug: node-js
  github:
    branch: production
    deploy_on_push: true
    repo: Digital-Grinnell/npm-rootstalk
  name: npm-rootstalk
  source_dir: /

It’s heavenly! And that’s all for now.

First Matomo Update

Mark McFate — Mon, 15 May 2023 18:33:16 +0000

Been prompted to do this for a couple of weeks so I finally took-the-plunge. https://analytics.summittservices.com/index.php?module=CoreUpdater&action=newVersionAvailable.

Wow, that was way too easy. I wonder if it really worked?

A Pagefind Problem?

Mark McFate — Wed, 10 May 2023 14:33:09 +0000

Not Just a Hugo Issue

Take note of the question mark at the end of the title, otherwise it could be somewhat misleading. This is not really a problem with Hugo, but one with cloud deployment of Hugo static apps, particularly as an Azure Static Web App.

The Nutshell

As you may know from post 143, I have successfully installed and configured Pagefind in Rootstalk, but thus far it only works locally. When I try to deploy Pagefind to the cloud, specifically as an Azure Static Web App, I can’t make it work because there’s no apparent way to invoke the necessary npx pagefind... command AFTER Hugo compiles the site, but BEFORE the site gets deployed. Azure leverages GitHub Actions to build Hugo sites, but that process also involves some custom/proprietary Azure scripts. Therein lies the problem.

The Azure script auto-detects the presence of a Hugo project, or any one of many other platforms. That makes the process super-easy to use, but nearly impossible to “customize”. Azure does provide some configuration variables to influence the script behavior, most notably there’s an app_build_command variable that looks promising, but it only works for node builds, and not for Hugo.

Three Possible Solutions

I can reasonably see three possible solutions to this dilema. In order of simplicity they are:

1. Skip Azure and Deploy Only to DigitalOcean

Ultimately, Rootstalk’s production instance is a DigitalOcean (DO) static web app, Azure is only used for “staging” of a locally viable Rootstalk instance. DO doesn’t use GitHub Actions, a drawback in my book, so its build script, called an “App Spec” in DO, is a little more “open”. The critical portion of the App Spec I use to deploy Rootstalk in DO reads like this:

static_sites:
- build_command: hugo -d public

That’s it, just a simple hugo -d public command. So, in theory I should be able to just change that to read:

static_sites:
- build_command: hugo -d public && npm_config_yes=true npx pagefind --source "public" --bundle-dir ../static/_pagefind

It’s basically just a “compound” command to run npx immediately after hugo is done compiling.

The problem with this very simple solution, assuming it even works, is that it’s somewhat unique to Rootstalk which is already deployed to DigitalOcean. Other Hugo projects, and I have many like this blog, that need Azure or another cloud provider other than DO, would not benefit from this fix.

Update 12-May-2023: Same Problem Exists in DigitalOcean

Well, I tried. I attempted to push my Pagefind additions to Rootstalk forward directly into production on DigitalOcean. Unfortunately, that failed in similar fashion to what I saw in Azure. It all works locally, but DigitalOcean, like Azure, has no way of “detecting” that my Hugo project has an NPM component, and therefore takes no steps to build NPM or Pagefind into my solution.

So, I’m moving on to option #2 below. Wish me luck.

2. Wrap Hugo in `npm`

So, node and npm seem to be all the rage these days, and perhaps for good reason. I recently fell in love with Eleventy/11ty because it’s Javascript, not Go, and it’s elegantly simple with tons of flexibility. If my Azure Static Web App was framed in node.js, as both Eleventy and Pagefind are, there would be no problem. The Azure scripts used to deploy those frameworks are far more customizable than Hugo, and there’s documentation to prove it.

So, how might I approach something like this? Well, A Powerful Blog Setup with Hugo and NPM by Tom Hombergs looks like a promising place to start. The process that Tom advocates leverages a neat little package called hugo-bin.

If strategy #1 fails I’ll certainly give this #2 a try. Even if #1 works, I might give this strategy a spin if/when I re-tool this blog.

Update 29-May-2023: NPM is GREAT!

Since my first option (above) failed very quickly, I moved on to this option and I honestly think I might transition every static site that I have to be wrapped inside NPM, it works wonderfully and is a really attractive solution for so many ills!

I’m going to document my experience with NPM wrapped around Hugo and will post that document here soon, probably as Wrap EVERYTHING in NPM!.

3. Transition from Hugo to Eleventy

Reasonable? Yes. A lot of work? That’s relative. The right thing to do? Probably.

So, it’s already been done, see This Blog in Eleventy + Ghost and Searching for a Search Solution. That’s all I’m going to say about this option, a definite maybe.

That’s all folks… for now.

Wrapping Hugo in NPM

Mark McFate — Tue, 09 May 2023 23:13:09 +0000

I ran into a big road block with implementation of Pagefind in Rootstalk today… everything works fine in development, but I can’t easily deploy to Azure because there’s no way to “inject” Pagefind into an Azure Static Web App build before the “public” content gets deployed. I can generate the Pagefind parts after deployment, but that does me no good.

Tomorrow I need to have a look at my Rootstalk DigitalOcean deployment to see if what I already have might work there (DO uses a build script that I can add an npx... command to). If that fails I need to look back at https://www.blogtrack.io/blog/powerful-blog-setup-with-hugo-and-npm/ to see if there’s a solution there for me.

-/-/-/

location: Toledo, IA

Searching for a Search Solution

Mark McFate — Thu, 20 Apr 2023 17:02:00 +0000

A few months ago I sat in on a CFE.dev webinar and I was really impressed with what I saw. I made a note to come back and look closer at Pagefind, and my new blog – this blog that you’re presumably reading right now – really needed a search feature, so the “Search” box on this page (I hope it’s there) is the outcome.

Glad I Found Pagefind

So, I started looking closely at search options and considered things like Lunr, which I’ve used before, and Solr, which I both love and hate (because of its JAVA roots). Pagefind was, of course, also on that short list and it quickly solidified its position at the top of the list when I found Adding search to an Eleventy site by Mike.

Mike doesn’t explain the setup in detail, but this comment really resonated with me so I decided to follow the same path:

Weirdly, I’ve found that I seem to read the posts on this site more than I write them. I say read, but I just mean referring to the stuff I’ve written because it has left my head.

Mike does provide a link to the reference he used, namely Using PageFind with Eleventy for Search by Robb Knight. I found Robb’s guidance and experience to be VERY helpful. Thank you, Robb!

Implementing Robb’s Approach

I won’t bother duplicating what Robb explains quite well, but this wasn’t a 10-minute solution for me, probably because I’m really still learning Javascript and Eleventy, and also because I’m using a Ghost back-end with packages managed via Yarn rather than npm.

So, in the few sections that follow I’ll try to briefly described what’s different about my approach versus Robb’s, along with what didn’t, and what ultimately did work.

`yarn`, Not `npm`

The first significant difference I encountered was the requirement that this blog project uses yarn, not npm for package management. Eventually I found that I just needed to replace Robb’s npm install pagefind command with a yarn add pagefind equivalent. I don’t really understand the details of that difference, but it worked.

eleventy.after Never Fires?

The biggest hurdle for me was trying to get this bit of Robb’s code to work.

const { execSync } = require('child_process')

module.exports = function(eleventyConfig) {
  eleventyConfig.on('eleventy.after', () => {
    execSync(`npx pagefind --source _site --glob \"**/*.html\"`, { encoding: 'utf-8' })
  })
}

In my case the --source parameter needed to be dist since this blog is generated into a directory by that name, and my configuration object is simply config rather than eleventyConfig. I made those simple changes and added my snippet to the .eleventy.js file, but found that it never triggered. I even replaced the execSync function call with a simple console.log statement and still got nothing. And still to this day, I have no idea why that didn’t work.

That snippet of code is supposed to be responsible for indexing all of the .html files after Eleventy has generated them. Well, I found out that everything else was working properly, but the _pagefind bundle generated by that indexing operation was just “missing”, since the index never was created. That made me wonder what would happen if I “forced” the system to index my content in a different manner…and that wonder ultimately worked.

Using `npx pagefind...` to Index Content

Having studied some of the Pagefind documentation I wondered what adding some post-processing to my build workflow might do, something like npx pagefind --source dist? That ultimately worked, but not in that original form.

I found that when I ran yarn start to locally rebuild the site, there was no generated ./dist/_pagefind bundle present, so I got no “Search” user interface elements, but there was an empty <div> where that belonged. So, with my localhost instance still running I opened a new terminal and ran npx pagefind --source dist in the project directory. Voila! The ./dist/_pagefind bundle magically appeared and the “Search” control was instantly visible in my localhost window! It was working, and it was beautiful.

Added a `pagefind` Script… Didn’t Work

Ok, with the above breakthrough in-hand I made this addition to the scripts: key in my package.json file:

  "pagefind": "npx pagefind --source dist"

Then I added the following to the end of my steps: key in .github/workflows/build-and-deploy.yml:

  - run: yarn pagefind

This worked to some extent, but wasn’t reliable and absolutely didn’t work when deployed to my DigitalOcean app. The inner-workings of yarn and build caching in these environments is still a DEEP mystery to me!

Extending My `encrypt` Script

What does work, both locally and on DigitalOcean, is essentially an “extension” of my old encrypt script. My old build workflow included these two elements of package.json and build-and-deploy.yml:

    "encrypt": "staticrypt ./dist/rebuild.html '***hidden***' --short -o ./dist/rebuild.html -f ./src/auth/login.html -t 'Summitt Dweller Blog - Rebuild' -i 'Please enter the passphrase.<br/>Hint: gh0st' --label-error 'Sorry, Ella says no. Try again.'"

…and…

  - run: yarn encrypt

I took note of the relative path spec for ./dist/rebuild.html in the encrypt command, reasoned that I should run pagefind in a similar fashion – but just before encryption – and came up with this change in package.json:

    "encrypt": "npx pagefind --source ./dist && staticrypt ./dist/rebuild.html '***hidden***' --short -o ./dist/rebuild.html -f ./src/auth/login.html -t 'Summitt Dweller Blog - Rebuild' -i 'Please enter the passphrase.<br/>Hint: gh0st' --label-error 'Sorry, Ella says no. Try again.'"

It works!

When I’m running locally my build-and-deploy.yml doesn’t apply so in that case I typically will do this IF I want my search index (and encryption) updated:

yarn start
yarn encrypt

When I rebuild in production on DigitalOcean my build-and-deploy.yml configuration IS run so including npx pagefind... as part of the yarn encrypt step does the trick.

Time to Test…

I’m going to save this post now and rebuild this blog on DigitalOcean, where with any luck, a search for a term like “DEEP” should return this post, and maybe one or two others. Drum roll please… Huzzah!

Man, Do I Feel Stupid

Previously I mentioned that my additon of a package.json script named pagefind didn’t work, and that my DigitalOcean builds kept trying to do yarn encrypt after yarn build. Well, turns out that’s because build-and-deploy.yml, where I had inserted the yarn pagefind step IS NOT in control of building my DigitalOcean app. In fact, it’s a .github workflow so I’m really not sure what role that file plays, if any?

I went looking in my DigitalOcean app configuration settings for yarn encrypt and found it, in the app’s “App Settings” subsection titled “App Spec”. When I view that “App Spec” I see this:

alerts:
- rule: DEPLOYMENT_FAILED
- rule: DOMAIN_FAILED
domains:
- domain: blog.summittdweller.com
  type: PRIMARY
  zone: summittdweller.com
envs:
- key: TZ
  scope: RUN_AND_BUILD_TIME
  value: America/Chicago
name: blog-summittdweller-11ty-ghost
region: nyc
static_sites:
- build_command: |-
    yarn build
    yarn encrypt
  environment_slug: node-js
  github:
    branch: main
    deploy_on_push: true
    repo: SummittDweller/blog-eleventy-ghost
  name: blog-eleventy-ghost
  output_dir: dist
  routes:
  - path: /
  source_dir: /

Note the two build_command: elements. Let’s change those two steps to three, like so:

- build_command: |-
    yarn build
    yarn pagefind
    yarn encrypt

Note that I captured the “App Spec” in a file named blog-summittdweller-11ty-ghost.yaml and I’ve saved that file in the project directory (where I hope it will do no harm). Attention: After making the aforementioned addition I almost forgot to upload this file back to DigitalOcean to replace our existing App Spec! Don’t be stupid like me!

Next, I needed a corresponding change in package.json, specifically these two lines (like I had them before, duh):

  "pagefind": "npx pagefind --source ./dist",
  "encrypt": "staticrypt ./dist/rebuild.html '***hidden***' --short -o ./dist/rebuild.html -f ./src/auth/login.html -t 'Summitt Dweller Blog - Rebuild' -i 'Please enter the passphrase.<br/>Hint: gh0st' --label-error 'Sorry, Ella says no. Try again.'"

Push to Rebuild

Ok, having modified and UPLOADEDblog-summittdweller-11ty-ghost.yaml, and having modified package.json, it came time to commit and push all my changes to see what happens. Another drum roll, please…

Implementing Pagefind Here

Mark McFate — Wed, 19 Apr 2023 05:38:32 +0000

Last evening I followed Using PageFind with Eleventy for Search to try and implement robust search capability in this blog. I’m working in a new pagefind branch of the code to pull this off.

Engaging `s3cmd` to Download and Resize Images from DO Spaces

Mark McFate — Sun, 16 Apr 2023 14:57:16 +0000

Earlier in this blog there’s a micropost that lays out the problem at hand. In a nutshell, most of the photos I saved in DigitalOcean Spaces are too big to be effective in this blog. They suck up lots of time and bandwidth every time this blog gets rebuilt, and then they are rendered at a fraction of their original size. This is exceptionall wasteful and unnecessary.

So, my task this morning is to install, configure, and engage the s3cmd tools to help get this done. There’s DigitalOcean-specific guidance provided in Setting Up s3cmd 2.x with DigitalOcean Spaces so that’s where I will begin. Here goes…

Installing `s3cmd`

This looks to be very simple and straightforward, brew install s3cmd should do the trick. In a substantially abridged form, that looked like this:

╭─mark@Marks-Mac-Mini ~/GitHub/blog-eleventy-ghost ‹main›
╰─$ brew install s3cmd
Running `brew update --auto-update`...
==> Auto-updated Homebrew!
==> Updated Homebrew from 4.0.10 (931327df1) to 4.0.13 (ac0663ae5).
Updated 4 taps (dart-lang/dart, sass/sass, homebrew/core and homebrew/cask).
...
You have 55 outdated formulae and 1 outdated cask installed.
...
==> Installing s3cmd
==> Pouring s3cmd--2.3.0.ventura.bottle.1.tar.gz
🍺  /usr/local/Cellar/s3cmd/2.3.0: 837 files, 11.4MB
==> Running `brew cleanup s3cmd`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
...

That took about 10 minutes, but should be worth it.

Configuration

Next, to begin configuration I need to fetch the access key:value pair for images-summittdweller that I created 4 years ago. I have the credentials necessary to view that information at https://cloud.digitalocean.com/account/api/spaces so that’s what I’ve done.

Whoa, that didn’t work! I tried to use old access keys but apparently they are in the wrong format? So, I created a new key named key-for-s3cmd and applied that (see below) and it works.

╭─mark@Marks-Mac-Mini ~/GitHub/blog-eleventy-ghost ‹main›
╰─$ s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key [images-summittdweller]: ***This is a secret***
Secret Key [4YPWYNXZZXFDIR6CRPHG]: ***An even bigger secret***
Default Region [US]:

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [nyc3.digitaloceanspaces.com]:

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.nyc3.digitaloceanspaces.com]:

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password [SummittDweller]:
Path to GPG program [/usr/local/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]:

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:

New settings:
  Access Key: ***This is a secret***
  Secret Key: ***An even bigger secret***
  Default Region: US
  S3 Endpoint: nyc3.digitaloceanspaces.com
  DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.nyc3.digitaloceanspaces.com
  Encryption password: SummittDweller
  Path to GPG program: /usr/local/bin/gpg
  Use HTTPS protocol: True
  HTTP Proxy server name:
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] Y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)

Save settings? [y/N] y
Configuration saved to '/Users/mark/.s3cfg'

Using `s3cwd`

The “More Information” section at the end of the document I used above says that… “You can use our quick reference on s3cmd usage to get started.” Ok, let’s have a look… very nice! I’m moving immediately to the Save files to your local computer bits.

Fetching Files of the August 2020 Derecho

The photos I saved of damage from our August 2020 Derecho, in a images-summittdweller/2020-Aug-10-Derecho bucket, are rather depressing and certainly don’t need to be super high-res for the web, so let’s retrieve those and practice this workflow there first.

The quick reference linked above says to use a command of the form s3cmd get s3://spacename/path/ --recursive to get all files from a particular folder. In my case, that’s…

╭─mark@Marks-Mac-Mini ~/Downloads/2020-derecho-photos
╰─$ s3cmd get s3://images-summittdweller/2020-Aug-10-Derecho/ --recursive

WARNING: Empty object name on S3 found, ignoring.
download: 's3://images-summittdweller/2020-Aug-10-Derecho/Derecho - 1.png' -> './Derecho - 1.png'  [1 of 82]
 23486944 of 23486944   100% in    1s    21.04 MB/s  done
download: 's3://images-summittdweller/2020-Aug-10-Derecho/Derecho - 10.png' -> './Derecho - 10.png'  [2 of 82]
 20861132 of 20861132   100% in    0s    23.45 MB/s  done
...
download: 's3://images-summittdweller/2020-Aug-10-Derecho/Photo-2020-08-20-19-51.jpeg' -> './Photo-2020-08-20-19-51.jpeg'  [81 of 82]
 6277783 of 6277783   100% in    0s    16.86 MB/s  done
ERROR: Creation of file './for-the-web/' failed (Reason: Is a directory)

Ok, 81 images downloaded. That was sooooooooo much better than the 1000-or-so clicks it would have taken to do that manually! The only gottcha… that last error message about not downloading the for-the-web directory, which is currently empty anyway. Before I go farther, I’m going to open that DigitalOcean Spaces bucket and change the errant directory name to web-ready; that’s where my “thumbnail” copies of all these images will be uploaded shortly.

Next Step, Resize Those Images

I found some concise and timely advice in the first web post I Googled: How to quickly resize multiple images at once on Mac. It uses exactly the tool I had in mind, the Mac-native Preview app.

Before begining that process I wanted to see just how much space my 81 images occupy, like so:

╭─mark@Marks-Mac-Mini ~/Downloads/2020-derecho-photos
╰─$ du -sh
2.7G	.

2.7G isn’t all that much, but it’s still much bigger than it needs to be. I’m firing up Preview now to see if I can reduce all images proportionally to a maximum width of 800 pixels. Be back in a couple of minutes…

Drum roll please…

╭─mark@Marks-Mac-Mini ~/Downloads/2020-derecho-photos
╰─$ du -sh
163M	.

Not bad at all. Only took a couple of minutes and it reduced the total image size from 2.7G down to 163M. Not too shabby.

Uploads Complete… Next?

In order to upload the resized images back to DigitalOcean Spaces, I simply used the nice graphical interface available to me at https://cloud.digitalocean.com/spaces/images-summittdweller?path=2020-Aug-10-Derecho%2Fweb-ready and the Upload option there. It worked nicely, but for whatever reason two of the images failed to upload: Derecho - 38.png and Derecho - 39.png. After a little review, and retries, I could not figure out why these wouldn’t upload but that’s OK because they are both largely redundant with Derecho - 40.png, so we’ll just stick with that one.

So, in the original folder image Derecho - 1.png has an endpoint of https://images.summittdweller.com/2020-Aug-10-Derecho/Derecho%20-%201.png. Makes sense. In my new web-ready directory the reduced-size copy of the same image naturally has an endpoint of https://images.summittdweller.com/2020-Aug-10-Derecho/web-ready/Derecho%20-%201.png. Fortunately, that’s a “match” except for the addition of the web-ready subdirectory in the path. Exactly what I hoped would happen.

This means the next step is to…

Alter Content to Include the `web-ready` Path

I used the admin search (magnifying glass) feature of the Ghost instance behind this blog to find all references to https://images.summittdweller.com/2020-Aug-10-Derecho and it found: NOTHING. Yikes. Turns out all of the references were in one post, and it was draft. So I edited the draft “manually” and published it. Now let’s try that search again… NOPE. So, the text inside those figure tags evidently isn’t discoverable. ☹️ Just as well since I think a different approach is needed.

Not `web-ready`, `originals`

Since editing the content (posts) to pick up the new web-ready directory is a royal pain, what I’m going to do now is create a new originals sub-directory in each bucket, download all the images, resize them locally as I did above, but replace the existing high-resolution images with the smaller copies AFTER I make a backup the original images in a new originals directory. That way I don’t have to edit any of my posts in order to get the smaller web-ready images, but I won’t lose the originals either. Beautimous.

Belay That Order!

While I’m confident this approach (above) will work, it may not be necessary. I looked, for example, in images-summittdweller/Norway-Photos-2019/Bergen - Hordaland, June 19, 2019 expecting to find some large images, but there were none over 800px in width. In fact, I believe they are all under 640px wide, so nothing to do there, and that’s a good thing.

That’s All Folks

A detailed search of images-summittdweller didn’t turn up any more enromous images that might get pushed to the web, so… “That’s All Folks”, for now at least.

Reduce Spaces Stored Image Sizes

Mark McFate — Sun, 16 Apr 2023 13:28:11 +0000

Ok, I get the sense that constantly pulling enormous digital images from DigitalOcean Spaces to rebuild this blog is unnecessarily chewing up lots of bandwidth. Today I’d like to begin fixing that by creating a utility, probably in Python, to grab all those images one-by-one, make a smaller copy of each, and put those copies back into a web-ready subdirectory in Spaces, or something like that.

There’s a wealth of probably pertinent information to be had in https://docs.digitalocean.com/reference/api/spaces-api/. Unfortunately, Spaces isn’t compatible with DO’s own command line utility, doctl, but it is compatible with s3cmd and the link I’ve provided documents how to get started with that. 🙂

Update: See Engaging s3cmd to Download and Resize Images from DO Spaces for my solution.

New GPX Merge

Mark McFate — Sat, 15 Apr 2023 13:42:58 +0000

Last evening I rode my bike to/from soccer so this morning I wanted to “merge” those two track logs before publishing them to https://hikes.SummittDweller.com. So, I found and forked https://github.com/locked-fg/gpxmerger to https://github.com/SummittDweller/gpxmerger and after tweaking my own code IT WORKS!

The command I ran to make my first “merge” was: python3 gpxmerger.py "/Users/mark/Downloads/2023-04-14 16:32 - 3m 11s.gpx" "/Users/mark/Downloads/2023-04-14 20:40 - 3m 27s.gpx". I think I might tweak the script soon to just pull all of the .gpx files from a specified directory.

Update: My fork of the project now accepts a single target_directory argument and uses “glob” to grab all the .gpx files in that directory.

Maybe Paperless NGX

Mark McFate — Wed, 12 Apr 2023 03:42:01 +0000

Today’s income tax prep adventures has me believing that I need a new instance of Paperless, maybe Paperless-NGX this time? Now if I can just get Mackenzie to stand up and configure a new bare-metal Linux server. 🙃

Updated Rootstalk with Landscape Orientation Message

Mark McFate — Mon, 10 Apr 2023 16:59:19 +0000

I pushed some Rootstalk updates into production (https://Rootstalk.Grinnell.edu) this morning and am making note of that here since I don’t have this slick micropost capability on my professional blog. 🙃

Rename Azure Subscription

Mark McFate — Sat, 04 Feb 2023 09:36:17 -0600

Today I plan to follow the guidance found in Rename Azure Subscriptions and Find Your Environments Faster to fix the name of my personal Azure subscription 1, and maybe more. That’s a horrible name, I know!

Done! The new subscription name is SummittDweller Pay-As-You-Go. Much better!

@Summitt Dweller’s Personal Blog - Development

Rebranding CV to HAT with Pagefind

A Newer Pagefind Enabled Blog!

Once Again…That Was Too Easy!

Following Tom’s Excellent Advice

It Works!

A Staging Site in Azure

Errors?

Last of the Errors?

Success!

A New Production Branch

A New Pagefind Enabled Blog!

That Was Too Easy!

Following Tom’s Excellent Advice

It Works!

A Staging Site in Azure

Errors?

Last of the Errors?

Success!

A New Production Branch

Spinning Up StaticCMS

The Plan

Along the Way

First Steps - Start with a Template

Getting Started

Let’s Go Local!

Eureka! It Works!

Even Better News?

Fresh Restart

Copying wieting-ss to static-cms-hugo-local-template

Corrections Needed

Time To Save My Progress

wieting-staticCMS

Time for a New Static CMS Approach

Improving My Azure Blob Storage

Building My First Steamlit App

Upgrading Matomo PHP

Streamlit Apps Fork

Fixing Wieting Guild Pages

Streamlit Ghost Azure Resources

Wrap EVERYTHING in NPM!

That Was Too Easy!

Following Tom’s Excellent Advice

It Works!

A Staging Site in Azure

A New Production Branch

First Matomo Update

A Pagefind Problem?

Not Just a Hugo Issue

The Nutshell

Three Possible Solutions

1. Skip Azure and Deploy Only to DigitalOcean

Update 12-May-2023: Same Problem Exists in DigitalOcean

2. Wrap Hugo in npm

Update 29-May-2023: NPM is GREAT!

3. Transition from Hugo to Eleventy

Wrapping Hugo in NPM

Searching for a Search Solution

Glad I Found Pagefind

Implementing Robb’s Approach

yarn, Not npm

eleventy.after Never Fires?

Using npx pagefind... to Index Content

Added a pagefind Script… Didn’t Work

Extending My encrypt Script

Time to Test…

Man, Do I Feel Stupid

Push to Rebuild

Implementing Pagefind Here

Engaging `s3cmd` to Download and Resize Images from DO Spaces

Installing s3cmd

Configuration

Using s3cwd

Fetching Files of the August 2020 Derecho

Next Step, Resize Those Images

Uploads Complete… Next?

Alter Content to Include the web-ready Path

Not web-ready, originals

Belay That Order!

That’s All Folks

Copying `wieting-ss` to `static-cms-hugo-local-template`

2. Wrap Hugo in `npm`

`yarn`, Not `npm`

Using `npx pagefind...` to Index Content

Added a `pagefind` Script… Didn’t Work

Extending My `encrypt` Script

Installing `s3cmd`

Using `s3cwd`

Alter Content to Include the `web-ready` Path

Not `web-ready`, `originals`