@Summitt Dweller’s Personal Blog - DigitalOcean

Upgrading Matomo PHP

Mark McFate — Mon, 12 Jun 2023 19:30:18 +0000

This afternoon I’m adding https://digital.grinnell.edu (site ID is 17) to my own Matomo instance and I’m prompted to upgrade PHP on that Matomo droplet (DigitalOcean). I did so quite easily following Upgrade PHP from 7.2 to 7.4 on CentOS 8 and Apache. I’ve also updated my Drupal version, and some modules, in https://digital.grinnell.edu along with adding the matomo module, but that module is now reporting:

    The validation of "http://analytics.summittservices.com/matomo.php" failed with error "Error opening socket ssl://analytics.summittservices.com:443" (HTTP code 0).
    The validation of "https://analytics.summittservices.com/matomo.php" failed with error "Error opening socket ssl://analytics.summittservices.com:443" (HTTP code 0).

Hmmm…wonder what’s up with that?

So, I tried this…

Last login: Mon Jun 12 19:21:45 2023 from 162.243.190.66
[root@centos-s-1vcpu-1gb-nyc1-01 ~]# sudo certbot renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/summittservices.com-0001.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for summittservices.com and analytics.summittservices.com
Reloading nginx server after certificate renewal

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/summittservices.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for summittservices.com and www.summittservices.com
Reloading nginx server after certificate renewal

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations, all renewals succeeded: 
  /etc/letsencrypt/live/summittservices.com-0001/fullchain.pem (success)
  /etc/letsencrypt/live/summittservices.com/fullchain.pem (success)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

…and… NO more validation errors! Yay!

Wrap EVERYTHING in NPM!

Mark McFate — Fri, 19 May 2023 22:27:00 +0000

So, node and npm seem to be all the rage these days, and perhaps for good reason. I recently fell in love with Eleventy/11ty over Hugo because it’s Javascript, not Go, and it’s elegantly simple with tons of flexibility. I recently tried to add Pagefind search to a Hugo static web site (see https://static.grinnell.edu/dlad-blog/posts/143-significant-rootstalk-retooling/). If Rootstalk, an Azure Static Web App was framed in node.js, as both Eleventy and Pagefind are, there would be no problem. The Azure scripts used to deploy those frameworks are far more customizable than Hugo, and there’s documentation to prove it.

So, how might I approach combining Hugo with Pagefind in the cloud? Well, A Powerful Blog Setup with Hugo and NPM* by Tom Hombergs looked like a promising place to start. The process that Tom advocates leverages a neat little package called hugo-bin.

*The links provided above and below are to a Wayback Machine capture of the original post.

That Was Too Easy!

Wrapping my Hugo site in an NPM package was so easy that I forgot to document it… and maybe I really didn’t need too. Time to catch up, so here’s a brief sysnopsis of what I did…

Following Tom’s Excellent Advice

I started by studying A Powerful Blog Setup with Hugo and NPM and quickly found that I could easily follow it almost verbatim. Since I already had npm and Hugo installed I skipped ahead to the section titled “Setting Up a Hugo Project” and then to the npx and git commands there. My experience, mostly in commands, looked like this on my Mac Mini workstation.

cd ~/GitHub/
# Named the new project npm-rootstalk so it would not confilct with any of my existing `rootstalk...` directories
mkdir npm-rootstalk      
cd npm-rootstalk
npx hugo new site . --force
# Copied the contents of my old project into the new one and then removed whatever I no longer needed
cp -fr ~/GitHub/rootstalk/. .  
# Installed Pagefind...
npm install pagefind
git init
# I created a new empty repo in GitHub called npm-rootstalk, then...
git remote add origin https://github.com/Digital-Grinnell/npm-rootstalk 
git add . 
git commit -m "Initial commit of new npm-wrapped Hugo project"
git push

Having secured a new repo I moved to running Hugo locally, then polishing my package.json scripts as Tom did. I now have a package.json that reads like this:

{
  "name": "npm-rootstalk",
  "version": "1.0.0",
  "description": "Rootstalk website built with Hugo and the Lightbi theme inside an NPM package.  May 2023.",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "build": "npm run hugo:build && npx pagefind --source public",
    "do:build": "hugo -d public && npx pagefind --source public",
    "azure:build": "hugo -d public --baseURL=\"https://yellow-wave-0e513e510.3.azurestaticapps.net/\" && npx pagefind --source public",
    "clean": "npm run hugo:clean",
    "serve": "npm run hugo:build && npx pagefind --source public --bundle-dir ../static/_pagefind && npm run hugo:serve",
    "hugo:build": "hugo -d public",
    "hugo:serve": "hugo server",
    "hugo:clean": "rm -rf build resources public"
  },
  "author": "Mark A. McFate",
  "license": "ISC",
  "dependencies": {
    "hugo-bin": "^0.102.0",
    "pagefind": "^0.12.0"
  }
}

It Works!

Now I just use npm run serve to build the Hugo site, index it with Pagefind, and serve it locally at http://localhost:1313.

A Staging Site in Azure

I can simply push changes to the main branch of the npm-rootstalk repo to trigger an Azure (aka GitHub Actions) rebuild, indexing and deployment of the site to this address.

A New Production Branch

Pushing to production is just as simple, I just have to push changes to the new production branch of the code and my app spec at DigitalOcean (see below) takes care of building, indexing, and deploying to https://rootstalk.grinnell.edu.

That DO app spec reads like this:

alerts:
- rule: DEPLOYMENT_FAILED
- rule: DOMAIN_FAILED
domains:
- domain: rootstalk.grinnell.edu
  type: PRIMARY
- domain: prairiejournal.grinnell.edu
  type: ALIAS
envs:
- key: TZ
  scope: RUN_AND_BUILD_TIME
  value: America/Chicago
- key: HUGO_MATOMO_ID
  scope: RUN_AND_BUILD_TIME
  value: "15"
ingress:
  rules:
  - component:
      name: npm-rootstalk
    match:
      path:
        prefix: /
name: npm-rootstalk
region: nyc
static_sites:
- build_command: npm run build
  environment_slug: node-js
  github:
    branch: production
    deploy_on_push: true
    repo: Digital-Grinnell/npm-rootstalk
  name: npm-rootstalk
  source_dir: /

It’s heavenly! And that’s all for now.

A Pagefind Problem?

Mark McFate — Wed, 10 May 2023 14:33:09 +0000

Not Just a Hugo Issue

Take note of the question mark at the end of the title, otherwise it could be somewhat misleading. This is not really a problem with Hugo, but one with cloud deployment of Hugo static apps, particularly as an Azure Static Web App.

The Nutshell

As you may know from post 143, I have successfully installed and configured Pagefind in Rootstalk, but thus far it only works locally. When I try to deploy Pagefind to the cloud, specifically as an Azure Static Web App, I can’t make it work because there’s no apparent way to invoke the necessary npx pagefind... command AFTER Hugo compiles the site, but BEFORE the site gets deployed. Azure leverages GitHub Actions to build Hugo sites, but that process also involves some custom/proprietary Azure scripts. Therein lies the problem.

The Azure script auto-detects the presence of a Hugo project, or any one of many other platforms. That makes the process super-easy to use, but nearly impossible to “customize”. Azure does provide some configuration variables to influence the script behavior, most notably there’s an app_build_command variable that looks promising, but it only works for node builds, and not for Hugo.

Three Possible Solutions

I can reasonably see three possible solutions to this dilema. In order of simplicity they are:

1. Skip Azure and Deploy Only to DigitalOcean

Ultimately, Rootstalk’s production instance is a DigitalOcean (DO) static web app, Azure is only used for “staging” of a locally viable Rootstalk instance. DO doesn’t use GitHub Actions, a drawback in my book, so its build script, called an “App Spec” in DO, is a little more “open”. The critical portion of the App Spec I use to deploy Rootstalk in DO reads like this:

static_sites:
- build_command: hugo -d public

That’s it, just a simple hugo -d public command. So, in theory I should be able to just change that to read:

static_sites:
- build_command: hugo -d public && npm_config_yes=true npx pagefind --source "public" --bundle-dir ../static/_pagefind

It’s basically just a “compound” command to run npx immediately after hugo is done compiling.

The problem with this very simple solution, assuming it even works, is that it’s somewhat unique to Rootstalk which is already deployed to DigitalOcean. Other Hugo projects, and I have many like this blog, that need Azure or another cloud provider other than DO, would not benefit from this fix.

Update 12-May-2023: Same Problem Exists in DigitalOcean

Well, I tried. I attempted to push my Pagefind additions to Rootstalk forward directly into production on DigitalOcean. Unfortunately, that failed in similar fashion to what I saw in Azure. It all works locally, but DigitalOcean, like Azure, has no way of “detecting” that my Hugo project has an NPM component, and therefore takes no steps to build NPM or Pagefind into my solution.

So, I’m moving on to option #2 below. Wish me luck.

2. Wrap Hugo in `npm`

So, node and npm seem to be all the rage these days, and perhaps for good reason. I recently fell in love with Eleventy/11ty because it’s Javascript, not Go, and it’s elegantly simple with tons of flexibility. If my Azure Static Web App was framed in node.js, as both Eleventy and Pagefind are, there would be no problem. The Azure scripts used to deploy those frameworks are far more customizable than Hugo, and there’s documentation to prove it.

So, how might I approach something like this? Well, A Powerful Blog Setup with Hugo and NPM by Tom Hombergs looks like a promising place to start. The process that Tom advocates leverages a neat little package called hugo-bin.

If strategy #1 fails I’ll certainly give this #2 a try. Even if #1 works, I might give this strategy a spin if/when I re-tool this blog.

Update 29-May-2023: NPM is GREAT!

Since my first option (above) failed very quickly, I moved on to this option and I honestly think I might transition every static site that I have to be wrapped inside NPM, it works wonderfully and is a really attractive solution for so many ills!

I’m going to document my experience with NPM wrapped around Hugo and will post that document here soon, probably as Wrap EVERYTHING in NPM!.

3. Transition from Hugo to Eleventy

Reasonable? Yes. A lot of work? That’s relative. The right thing to do? Probably.

So, it’s already been done, see This Blog in Eleventy + Ghost and Searching for a Search Solution. That’s all I’m going to say about this option, a definite maybe.

That’s all folks… for now.

Wrapping Hugo in NPM

Mark McFate — Tue, 09 May 2023 23:13:09 +0000

I ran into a big road block with implementation of Pagefind in Rootstalk today… everything works fine in development, but I can’t easily deploy to Azure because there’s no way to “inject” Pagefind into an Azure Static Web App build before the “public” content gets deployed. I can generate the Pagefind parts after deployment, but that does me no good.

Tomorrow I need to have a look at my Rootstalk DigitalOcean deployment to see if what I already have might work there (DO uses a build script that I can add an npx... command to). If that fails I need to look back at https://www.blogtrack.io/blog/powerful-blog-setup-with-hugo-and-npm/ to see if there’s a solution there for me.

-/-/-/

location: Toledo, IA

Searching for a Search Solution

Mark McFate — Thu, 20 Apr 2023 17:02:00 +0000

A few months ago I sat in on a CFE.dev webinar and I was really impressed with what I saw. I made a note to come back and look closer at Pagefind, and my new blog – this blog that you’re presumably reading right now – really needed a search feature, so the “Search” box on this page (I hope it’s there) is the outcome.

Glad I Found Pagefind

So, I started looking closely at search options and considered things like Lunr, which I’ve used before, and Solr, which I both love and hate (because of its JAVA roots). Pagefind was, of course, also on that short list and it quickly solidified its position at the top of the list when I found Adding search to an Eleventy site by Mike.

Mike doesn’t explain the setup in detail, but this comment really resonated with me so I decided to follow the same path:

Weirdly, I’ve found that I seem to read the posts on this site more than I write them. I say read, but I just mean referring to the stuff I’ve written because it has left my head.

Mike does provide a link to the reference he used, namely Using PageFind with Eleventy for Search by Robb Knight. I found Robb’s guidance and experience to be VERY helpful. Thank you, Robb!

Implementing Robb’s Approach

I won’t bother duplicating what Robb explains quite well, but this wasn’t a 10-minute solution for me, probably because I’m really still learning Javascript and Eleventy, and also because I’m using a Ghost back-end with packages managed via Yarn rather than npm.

So, in the few sections that follow I’ll try to briefly described what’s different about my approach versus Robb’s, along with what didn’t, and what ultimately did work.

`yarn`, Not `npm`

The first significant difference I encountered was the requirement that this blog project uses yarn, not npm for package management. Eventually I found that I just needed to replace Robb’s npm install pagefind command with a yarn add pagefind equivalent. I don’t really understand the details of that difference, but it worked.

eleventy.after Never Fires?

The biggest hurdle for me was trying to get this bit of Robb’s code to work.

const { execSync } = require('child_process')

module.exports = function(eleventyConfig) {
  eleventyConfig.on('eleventy.after', () => {
    execSync(`npx pagefind --source _site --glob \"**/*.html\"`, { encoding: 'utf-8' })
  })
}

In my case the --source parameter needed to be dist since this blog is generated into a directory by that name, and my configuration object is simply config rather than eleventyConfig. I made those simple changes and added my snippet to the .eleventy.js file, but found that it never triggered. I even replaced the execSync function call with a simple console.log statement and still got nothing. And still to this day, I have no idea why that didn’t work.

That snippet of code is supposed to be responsible for indexing all of the .html files after Eleventy has generated them. Well, I found out that everything else was working properly, but the _pagefind bundle generated by that indexing operation was just “missing”, since the index never was created. That made me wonder what would happen if I “forced” the system to index my content in a different manner…and that wonder ultimately worked.

Using `npx pagefind...` to Index Content

Having studied some of the Pagefind documentation I wondered what adding some post-processing to my build workflow might do, something like npx pagefind --source dist? That ultimately worked, but not in that original form.

I found that when I ran yarn start to locally rebuild the site, there was no generated ./dist/_pagefind bundle present, so I got no “Search” user interface elements, but there was an empty <div> where that belonged. So, with my localhost instance still running I opened a new terminal and ran npx pagefind --source dist in the project directory. Voila! The ./dist/_pagefind bundle magically appeared and the “Search” control was instantly visible in my localhost window! It was working, and it was beautiful.

Added a `pagefind` Script… Didn’t Work

Ok, with the above breakthrough in-hand I made this addition to the scripts: key in my package.json file:

  "pagefind": "npx pagefind --source dist"

Then I added the following to the end of my steps: key in .github/workflows/build-and-deploy.yml:

  - run: yarn pagefind

This worked to some extent, but wasn’t reliable and absolutely didn’t work when deployed to my DigitalOcean app. The inner-workings of yarn and build caching in these environments is still a DEEP mystery to me!

Extending My `encrypt` Script

What does work, both locally and on DigitalOcean, is essentially an “extension” of my old encrypt script. My old build workflow included these two elements of package.json and build-and-deploy.yml:

    "encrypt": "staticrypt ./dist/rebuild.html '***hidden***' --short -o ./dist/rebuild.html -f ./src/auth/login.html -t 'Summitt Dweller Blog - Rebuild' -i 'Please enter the passphrase.<br/>Hint: gh0st' --label-error 'Sorry, Ella says no. Try again.'"

…and…

  - run: yarn encrypt

I took note of the relative path spec for ./dist/rebuild.html in the encrypt command, reasoned that I should run pagefind in a similar fashion – but just before encryption – and came up with this change in package.json:

    "encrypt": "npx pagefind --source ./dist && staticrypt ./dist/rebuild.html '***hidden***' --short -o ./dist/rebuild.html -f ./src/auth/login.html -t 'Summitt Dweller Blog - Rebuild' -i 'Please enter the passphrase.<br/>Hint: gh0st' --label-error 'Sorry, Ella says no. Try again.'"

It works!

When I’m running locally my build-and-deploy.yml doesn’t apply so in that case I typically will do this IF I want my search index (and encryption) updated:

yarn start
yarn encrypt

When I rebuild in production on DigitalOcean my build-and-deploy.yml configuration IS run so including npx pagefind... as part of the yarn encrypt step does the trick.

Time to Test…

I’m going to save this post now and rebuild this blog on DigitalOcean, where with any luck, a search for a term like “DEEP” should return this post, and maybe one or two others. Drum roll please… Huzzah!

Man, Do I Feel Stupid

Previously I mentioned that my additon of a package.json script named pagefind didn’t work, and that my DigitalOcean builds kept trying to do yarn encrypt after yarn build. Well, turns out that’s because build-and-deploy.yml, where I had inserted the yarn pagefind step IS NOT in control of building my DigitalOcean app. In fact, it’s a .github workflow so I’m really not sure what role that file plays, if any?

I went looking in my DigitalOcean app configuration settings for yarn encrypt and found it, in the app’s “App Settings” subsection titled “App Spec”. When I view that “App Spec” I see this:

alerts:
- rule: DEPLOYMENT_FAILED
- rule: DOMAIN_FAILED
domains:
- domain: blog.summittdweller.com
  type: PRIMARY
  zone: summittdweller.com
envs:
- key: TZ
  scope: RUN_AND_BUILD_TIME
  value: America/Chicago
name: blog-summittdweller-11ty-ghost
region: nyc
static_sites:
- build_command: |-
    yarn build
    yarn encrypt
  environment_slug: node-js
  github:
    branch: main
    deploy_on_push: true
    repo: SummittDweller/blog-eleventy-ghost
  name: blog-eleventy-ghost
  output_dir: dist
  routes:
  - path: /
  source_dir: /

Note the two build_command: elements. Let’s change those two steps to three, like so:

- build_command: |-
    yarn build
    yarn pagefind
    yarn encrypt

Note that I captured the “App Spec” in a file named blog-summittdweller-11ty-ghost.yaml and I’ve saved that file in the project directory (where I hope it will do no harm). Attention: After making the aforementioned addition I almost forgot to upload this file back to DigitalOcean to replace our existing App Spec! Don’t be stupid like me!

Next, I needed a corresponding change in package.json, specifically these two lines (like I had them before, duh):

  "pagefind": "npx pagefind --source ./dist",
  "encrypt": "staticrypt ./dist/rebuild.html '***hidden***' --short -o ./dist/rebuild.html -f ./src/auth/login.html -t 'Summitt Dweller Blog - Rebuild' -i 'Please enter the passphrase.<br/>Hint: gh0st' --label-error 'Sorry, Ella says no. Try again.'"

Push to Rebuild

Ok, having modified and UPLOADEDblog-summittdweller-11ty-ghost.yaml, and having modified package.json, it came time to commit and push all my changes to see what happens. Another drum roll, please…

Engaging `s3cmd` to Download and Resize Images from DO Spaces

Mark McFate — Sun, 16 Apr 2023 14:57:16 +0000

Earlier in this blog there’s a micropost that lays out the problem at hand. In a nutshell, most of the photos I saved in DigitalOcean Spaces are too big to be effective in this blog. They suck up lots of time and bandwidth every time this blog gets rebuilt, and then they are rendered at a fraction of their original size. This is exceptionall wasteful and unnecessary.

So, my task this morning is to install, configure, and engage the s3cmd tools to help get this done. There’s DigitalOcean-specific guidance provided in Setting Up s3cmd 2.x with DigitalOcean Spaces so that’s where I will begin. Here goes…

Installing `s3cmd`

This looks to be very simple and straightforward, brew install s3cmd should do the trick. In a substantially abridged form, that looked like this:

╭─mark@Marks-Mac-Mini ~/GitHub/blog-eleventy-ghost ‹main›
╰─$ brew install s3cmd
Running `brew update --auto-update`...
==> Auto-updated Homebrew!
==> Updated Homebrew from 4.0.10 (931327df1) to 4.0.13 (ac0663ae5).
Updated 4 taps (dart-lang/dart, sass/sass, homebrew/core and homebrew/cask).
...
You have 55 outdated formulae and 1 outdated cask installed.
...
==> Installing s3cmd
==> Pouring s3cmd--2.3.0.ventura.bottle.1.tar.gz
🍺  /usr/local/Cellar/s3cmd/2.3.0: 837 files, 11.4MB
==> Running `brew cleanup s3cmd`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
...

That took about 10 minutes, but should be worth it.

Configuration

Next, to begin configuration I need to fetch the access key:value pair for images-summittdweller that I created 4 years ago. I have the credentials necessary to view that information at https://cloud.digitalocean.com/account/api/spaces so that’s what I’ve done.

Whoa, that didn’t work! I tried to use old access keys but apparently they are in the wrong format? So, I created a new key named key-for-s3cmd and applied that (see below) and it works.

╭─mark@Marks-Mac-Mini ~/GitHub/blog-eleventy-ghost ‹main›
╰─$ s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key [images-summittdweller]: ***This is a secret***
Secret Key [4YPWYNXZZXFDIR6CRPHG]: ***An even bigger secret***
Default Region [US]:

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [nyc3.digitaloceanspaces.com]:

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.nyc3.digitaloceanspaces.com]:

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password [SummittDweller]:
Path to GPG program [/usr/local/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]:

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:

New settings:
  Access Key: ***This is a secret***
  Secret Key: ***An even bigger secret***
  Default Region: US
  S3 Endpoint: nyc3.digitaloceanspaces.com
  DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.nyc3.digitaloceanspaces.com
  Encryption password: SummittDweller
  Path to GPG program: /usr/local/bin/gpg
  Use HTTPS protocol: True
  HTTP Proxy server name:
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] Y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Success. Encryption and decryption worked fine :-)

Save settings? [y/N] y
Configuration saved to '/Users/mark/.s3cfg'

Using `s3cwd`

The “More Information” section at the end of the document I used above says that… “You can use our quick reference on s3cmd usage to get started.” Ok, let’s have a look… very nice! I’m moving immediately to the Save files to your local computer bits.

Fetching Files of the August 2020 Derecho

The photos I saved of damage from our August 2020 Derecho, in a images-summittdweller/2020-Aug-10-Derecho bucket, are rather depressing and certainly don’t need to be super high-res for the web, so let’s retrieve those and practice this workflow there first.

The quick reference linked above says to use a command of the form s3cmd get s3://spacename/path/ --recursive to get all files from a particular folder. In my case, that’s…

╭─mark@Marks-Mac-Mini ~/Downloads/2020-derecho-photos
╰─$ s3cmd get s3://images-summittdweller/2020-Aug-10-Derecho/ --recursive

WARNING: Empty object name on S3 found, ignoring.
download: 's3://images-summittdweller/2020-Aug-10-Derecho/Derecho - 1.png' -> './Derecho - 1.png'  [1 of 82]
 23486944 of 23486944   100% in    1s    21.04 MB/s  done
download: 's3://images-summittdweller/2020-Aug-10-Derecho/Derecho - 10.png' -> './Derecho - 10.png'  [2 of 82]
 20861132 of 20861132   100% in    0s    23.45 MB/s  done
...
download: 's3://images-summittdweller/2020-Aug-10-Derecho/Photo-2020-08-20-19-51.jpeg' -> './Photo-2020-08-20-19-51.jpeg'  [81 of 82]
 6277783 of 6277783   100% in    0s    16.86 MB/s  done
ERROR: Creation of file './for-the-web/' failed (Reason: Is a directory)

Ok, 81 images downloaded. That was sooooooooo much better than the 1000-or-so clicks it would have taken to do that manually! The only gottcha… that last error message about not downloading the for-the-web directory, which is currently empty anyway. Before I go farther, I’m going to open that DigitalOcean Spaces bucket and change the errant directory name to web-ready; that’s where my “thumbnail” copies of all these images will be uploaded shortly.

Next Step, Resize Those Images

I found some concise and timely advice in the first web post I Googled: How to quickly resize multiple images at once on Mac. It uses exactly the tool I had in mind, the Mac-native Preview app.

Before begining that process I wanted to see just how much space my 81 images occupy, like so:

╭─mark@Marks-Mac-Mini ~/Downloads/2020-derecho-photos
╰─$ du -sh
2.7G	.

2.7G isn’t all that much, but it’s still much bigger than it needs to be. I’m firing up Preview now to see if I can reduce all images proportionally to a maximum width of 800 pixels. Be back in a couple of minutes…

Drum roll please…

╭─mark@Marks-Mac-Mini ~/Downloads/2020-derecho-photos
╰─$ du -sh
163M	.

Not bad at all. Only took a couple of minutes and it reduced the total image size from 2.7G down to 163M. Not too shabby.

Uploads Complete… Next?

In order to upload the resized images back to DigitalOcean Spaces, I simply used the nice graphical interface available to me at https://cloud.digitalocean.com/spaces/images-summittdweller?path=2020-Aug-10-Derecho%2Fweb-ready and the Upload option there. It worked nicely, but for whatever reason two of the images failed to upload: Derecho - 38.png and Derecho - 39.png. After a little review, and retries, I could not figure out why these wouldn’t upload but that’s OK because they are both largely redundant with Derecho - 40.png, so we’ll just stick with that one.

So, in the original folder image Derecho - 1.png has an endpoint of https://images.summittdweller.com/2020-Aug-10-Derecho/Derecho%20-%201.png. Makes sense. In my new web-ready directory the reduced-size copy of the same image naturally has an endpoint of https://images.summittdweller.com/2020-Aug-10-Derecho/web-ready/Derecho%20-%201.png. Fortunately, that’s a “match” except for the addition of the web-ready subdirectory in the path. Exactly what I hoped would happen.

This means the next step is to…

Alter Content to Include the `web-ready` Path

I used the admin search (magnifying glass) feature of the Ghost instance behind this blog to find all references to https://images.summittdweller.com/2020-Aug-10-Derecho and it found: NOTHING. Yikes. Turns out all of the references were in one post, and it was draft. So I edited the draft “manually” and published it. Now let’s try that search again… NOPE. So, the text inside those figure tags evidently isn’t discoverable. ☹️ Just as well since I think a different approach is needed.

Not `web-ready`, `originals`

Since editing the content (posts) to pick up the new web-ready directory is a royal pain, what I’m going to do now is create a new originals sub-directory in each bucket, download all the images, resize them locally as I did above, but replace the existing high-resolution images with the smaller copies AFTER I make a backup the original images in a new originals directory. That way I don’t have to edit any of my posts in order to get the smaller web-ready images, but I won’t lose the originals either. Beautimous.

Belay That Order!

While I’m confident this approach (above) will work, it may not be necessary. I looked, for example, in images-summittdweller/Norway-Photos-2019/Bergen - Hordaland, June 19, 2019 expecting to find some large images, but there were none over 800px in width. In fact, I believe they are all under 640px wide, so nothing to do there, and that’s a good thing.

That’s All Folks

A detailed search of images-summittdweller didn’t turn up any more enromous images that might get pushed to the web, so… “That’s All Folks”, for now at least.

Proper Python

Mark McFate — Sat, 24 Sep 2022 14:26:44 -0500

What follows is the README.md file from my GitHub repo where https://blog.summittdweller.com is built. It outlines how to “properly” add Python to a project such that we can create an easily reproducible “virtual environment” without storing the necessary Python libraries in GitHub.

From the README.md…

# blogs-SummittDweller

The Python portions of this repo were gleaned from https://github.com/SummittDweller/hikes. This is the repo for my https://blog.SummittDweller.com/ static website.

Latest Python Improvements

Using guidance at https://predictivehacks.com/how-to-work-with-vs-code-and-virtual-environments-in-python/ I successfully created a new .venv virtual environment. The commands I used to do this were…

╭─mark@Marks-Mac-Mini ~/GitHub/blogs-SummittDweller ‹main*›
╰─$ python3 -m venv .venv

╭─mark@Marks-Mac-Mini ~/GitHub/blogs-SummittDweller ‹main*›
╰─$ source .venv/bin/activate

(.venv) ╭─mark@Marks-Mac-Mini ~/GitHub/blogs-SummittDweller ‹main› 
╰─$ pip3 freeze > python-requirements.txt

(.venv) ╭─mark@Marks-Mac-Mini ~/GitHub/blogs-SummittDweller ‹main*›
╰─$ pip3 install -r python-requirements.txt

Then I used COMMAND-SHIFT-P to select and Python: Select interpreter to choose the new .venv Python. After that…

(.venv) ╭─mark@Marks-Mac-Mini ~/GitHub/blogs-SummitDweller ‹main*›
╰─$ python3 consolidate.py --start 20220806 --end 20220825

(.venv) ╭─mark@Marks-Mac-Mini ~/GitHub/blogs-SummitDweller ‹main*›
╰─$ hugo server

Rebuilding a Local Environment

Based on guideance from https://stackoverflow.com/questions/6590688/is-it-bad-to-have-my-virtualenv-directory-inside-my-git-repository I’ve reconstructed this project such that Python artifacts do NOT propogate into Github.

The project should automatically “rebuild” as necessary when opened in VSCode, but if the Python virtual environment needs to be recreated use…

source .venv/bin/activate && pip3 install -r python-requirements.txt

Deployment via DigitalOcean

In order to deploy this starter static app on DigitalOcean I had to PERMANENTLY move all of my Python components (.py scripts and requirements.txt) to a .python directory to hide it from view during app creation. Before moving these elements DigitalOcean would consistently try to deploy my app as Python, and that’s not right.

See DigitalOcean ticket [#06849069] Hugo static site with Python components builds fail for details.

The response I received from DigitalOcean…

Hello,

Thanks for contacting DigitalOcean.

We appreciate you reaching out to us about your Hugo static site. From what you mentioned it does sound like your app is getting detected as a python webservice rather then a static site due to the requirements.txt file you had in the root of your repo.

Unfortunately this is currently what App platform uses to detect applications as python apps. Either having a requirements.txt file or a setup.py file will cause that detection. Our engineers are looking into options to specify the build pack used rather then relying upon the auto detection, which would address this issue in your case, but don’t have a timeline yet for implementing it.

In the mean time the work around that you have specified here should work fine for this. It should cause the app to be detected as a static site rather then python. Hopefully it isn’t impacting anything else in your app’s build process.

If you have any questions, please let us know.

Regards, Nate Senior Cloud Support Engineer Check out our community for great tutorials, articles and FAQs! digitalocean.com/community

ref:_00Df218t5m._5004P21l4JM:ref

Remedy

Based on the response above I’m making final changes to eliminate the .python folder, renaming .python/requirements.txt as python-requirements.txt, and pulling all other contents of .python into the root of the project.