Projects and thoughts

This Site

I created this site as a hobby project to teach myself more about how to create and manage my own online presence, to learn more about the major changes in modern JavaScript (ex: promises), and to become familiar with cloud technologies.

Other than domain name registration (which I set up through a separate registrar) everything you see on this page is built and delivered using AWS CloudFormation, including:

  • Storage configured as basic web servers with public access
  • DNS zone and record management
  • CI/CD pipeline to automatically deploy changes to the site

The source for my CloudFormation template can be found on GitHub.

Home Environment Monitoring

After taking an introductory course to machine learning and using Tensorflow, it got me thinking about the value of data, and interaction between IoT and scalable compute available in cloud providers. I began to get interested in experimenting with using cheap sensors to gather lots of my own data, upload it to the cloud, and do...something with a it.

Choosing the hardware

I decided on the Raspberry Pi platform as the basis for my experimenting, and opted for the Raspberry Pi Zero W, which is the wireless version of the tiny Zero model and can be often picked up for only $15.

I then began to look into the various sensors available for the Pi Zero and options for connecting them. Most tutorials I found involved soldering components together, which was not something I was interested in. Happily, I came across a solderless solution to holds components together, and even an installation jig that was a perfect fit for Pi Zeros.

Once I settled on the board and how to connect sensors, I thought the first good sensor tests would be a camera and the Enviro pHAT environment sensor board, so I picked up those as well.

After everything arrived, I felt that the discrete data from the Enviro pHAT would be simpler to work with than the camera, so after I got the Pi running and Enviro pHAT attached, I began experimenting with the Enviro pHAT's Python library to acquire data from its sensors, and from it was able to get a sense of what values I wanted to keep, what to call them, and how to store them.

Raspberry Pi with solderless Enviro pHAT attachment
Raspberry Pi with solderless Enviro pHAT attachment.
Data storage

With an idea of what the data should look like, I then began to think about where to keep it. I still wasn't sure what I wanted to do with the data, but if I could at least store it somewhere, then I could work with it later when an idea strikes.

Remember, I wanted to explore the interaction between IoT and cloud services. Through a workplace acquisition, I became an employee at Oracle, thought it would be a natural fit to use OCI for this project.

I approached evaluating storage options for this environmental data the same way I did for this website - I wanted something simple and cheap. Generally when someone looks up database options in OCI, they usually see options such as Oracle Database (of course) and MySQL. For this purpose, though, I also stumbled across an option that isn't widely advertised - Oracle's NoSQL database for basic put and get operations.

With a database engine selected, I then had to figure out how to get data inside. The latest trend seemed to be with serverless micro-services, and since all I needed was basic put and get operations, developing them as serverless functions made sense. When it came to picking a runtime for the functions, I didn't have experience with Node.js, Ruby, or Go, so to continue with learning more about modern JavaScript, went with Node.

Once the Node functions were working on my machine, I started looking at how OCI actually deploys serverless functions, which is built on the open source Fn Project, which works somewhat differently than AWS's Lambda. Having prior experience with Lambda, OCI Functions took me a bit to wrap my brain around. The key difference for me to understand was Lambda (and AWS Serverless Application Model [SAM]) is a 'walled garden' - pick your runtime, upload your source code, and Lambda handles creating, hosting, and rolling out the container image for you.

The Fn Project on the other hand, is meant to be platform agnostic, and is therefore BYOC (Bring Your Own Container). While this does allow for much greater flexibility, and the project does have runtime-specific base images and SDK 'hooks' to ease development, it puts the ultimate onus for creating, hosting, and deploying the container image on the developer.

While the OCI Functions CLI can be used to build, upload, and deploy function images, invoking the CLI for deployments from my PC or the cloud shell was still a chore - not the seamless CI/CD experience I wanted. Therefore, I expanded on my previous experience with CI/CD pipeline creation in AWS and Jenkins, and applied it to this problem as well. OCI has several Operations-oriented with a free usage tier, including:

  • DevOps Build pipelines - executes the Fn commands to build the container images
  • Container Registries - stores the images after they're built
  • DevOps Deploy pipelines - pulls the images down and deploys them to pre-existing OCI Functions
  • Resource Manager - roughly equivalent to AWS CloudFormation, is a managed Terraform offering to track stack state

With these tools, I was able to create a Resource Manager stack that not only initially sets up the NoSQL database and OCI Functions, but also manages the Build pipeline and container registry to build the first image (required to initially seed an OCI Function), and the Deploy pipeline to automate function updates every time I push a commit - all in a declarative, source-controlled repo of its own.

DevOps Build pipeline to prepare NodeJS functions
DevOps Build pipeline to prepare NodeJS functions

The source for my Fn-compatible NoSQL-backed 'put' and 'get' functions, and build specification for use in the Build pipeline, can be found on GitHub.

Furthermore, the source for my Resource Manager Terraform module to create the pipeline, Functions environment, and database can be found on GitHub as well.

Turning on the tap

With new pipeline to deploy and refine my NoSQL database schema and OCI Functions, I went back to my Raspberry Pi to flesh out how to take the data from the Enviro pHAT Python library and send it to the OCI Function. Most of the instructions I found online refer to calling Functions through OCI's API Gateway, which sounded nice, until I saw how complicated authentication would be. Again, trying to keep things simple and cheap. I had to figure out another way to authenticate to this 'put' function from a Raspberry Pi without having to write my own authentication scheme.

Since only my Pi should be able to post data into my database, I expanded my Terraform module to handle creating a new IAM user, group, and policy which only allowed invoking the 'put' Function. I then manually generated an authentication certificate for that user, transferred it to my Pi, and in short order wired together the Enviro pHAT Python library and the OCI SDK for Python to upload the data from the sensors into the NoSQL database through OCI Functions.

Grid of NoSQL table with data successfully uploaded from Raspberry Pi
Success!

My Python code to capture sensor data and pass it to an OCI Function can be found on GitHub

Bringing it all together

After I got data flowing into NoSQL, I wondered - How could I graph this data? Using JavaScript to display the graph on this web page seemed like a natural fit, but I had no graphing experience in JavaScript and didn't know how I'd invoke the OCI function from an anonymous web page in order to pull the data out.

OCI's API Gateway was very straightforward to set up, especially for anonymous access, so I once again expanded my Terraform module to create an API gateway and link it to the 'get' Function that, in turn, was wired to the NoSQL database where this sensor data was being stored.

Once I was able to anonymously pull back a sample of data from my Function into my browser, I then experimented with a couple different JavaScript chart libraries. Eventually I settled on the combination of Chart.js and the Luxon date/time libraries due to Chart.js's support for having multiple y-axes, and Luxon's support for seamlessly presenting UTC timestamps in the browser's local timezone.

The result is below:

Windows Package Management

A project at work involved rebuilding legacy Citrix images on a new version of Windows Server created an opportunity for me to introduce a configuration management framework in our image build and update processes. I took this opportunity to learn more about PowerShell Desired State Configuration (DSC) as a means to manage these images (Chef on Windows wasn't quite where we needed it to be at the time).

Getting to know DSC

I began exploring the use of the 'package' DSC resource, but found it very clumsy to work with. It's expression in DSC configuration data didn't line up well how my team's engineers thought about the layered software we manage on our images. I began searching for alternatives, and came across another option - PowerShell PackageManagement, which exposed both a series of PowerShell cmdlets and DSC resources that could be used to both manage packages and package sources.

PackageManagement sounded like it might work well for my use case, so I began to investigate what options it had to manage the complex collection of 3rd party software we required, when I stumbled upon the ChocolateyGet package provider module.

I already had some (very limited) past experience with Chocolatey as a package management tool, but had not previously known that it could be managed through a PowerShell-based API, and being able to manage Chocolatey packages and package sources via DSC intrigued me, so I began to explore how I could leverage the ChocolateyGet provider to that end.

Updating ChocolateyGet

However, I encountered a huge initial roadblock, which was that while PackageManagement, as a framework, supported package source management and installing packages from custom sources, the ChocolateyGet provider did not. It was hard-coded to use the Chocolatey.org public community repository, and there were no plans by the module maintainer to add custom source support anytime soon.

Here was this module that did almost everything I needed - I was so close! So I decided to fork the module and make it my own.

This was my first real attempt at taking an open source project and trying to contribute to it, so I worked pretty diligently to leave the existing functionality and structure of the code base as-is and just add the missing features I needed to manage Chocolatey sources through DSC.

After a while, I finally got the provider managing, searching, and installing across sources in a way that was idiomatic to both PackageManagement and Chocolatey, and began to use it with DSC in real environments. However, that real-world use uncovered several other smaller problems after that, among which included:

  • faulty error propagation
  • package parameters not working with DSC
  • inability to dynamically 'upgrade' an installed package to the latest version
  • slow performance when downloading packages with embedded installers
  • large sections of duplicated (and sometimes confusing) logic that made code maintenance tedious and error-prone
The finished product

Working through these additional blockers and eventually using my modified version of the provider, we were able to successfully roll out a DSC-based framework for building and upgrading the Citrix images at work, and used that framework for a couple years (until we switched to Chef). Furthermore, I was really happy in general with how the provider worked, and thought would work well for other system administrators, too.

Since there hadn't been much activity on the ChocolateyGet project, I reached out to the module maintainer about being a project contributor and merged in my changes, which I later followed up with:

  • a major refactor of the module to deduplicate logic and improve readability
  • updating the CI/CD pipeline with better tests and automatic publishing
  • abstracting CLI interaction with PowerShell Crescendo into another module I own called Foil

Sample output of the ChocolateyGet CI/CD pipeline
CI/CD pipeline

After having effectively rewritten the module from the ground up, I've effectively become the primary maintainer of the ChocolateyGet project, and continue to review any issues or contributions from the community.

ChocolateyGet can be installed via PowerShell Gallery.

Along comes WinGet

When Microsoft announced their new native Windows package manager WinGet in May 2020 without an accompanying native PowerShell module, I saw a need in the community that I could help quickly address by taking the work I'd already done on ChocolateyGet to abstract and express package and repository management concepts, and adapting it to WinGet. Within a day of WinGet's announcement, I released a basic PackageManagement module to the community, though the underlying CLI it interacted with had several quirks and didn't work very well.

Unfortunately, further development, including reworking the CLI interaction into a PowerShell Crescendo module, was impeded due to it being practically impossible to install the CLI on any version of Windows Server at the time, thus ruling out any opportunity for automated testing.

It wasn't until about a year and a half later, after WinGet itself had matured somewhat and Windows Server 2022 (the first version of Server WinGet could even technically run on) runners became available with GitHub Actions, that I revisited the project. After much trail and error, I was able to automate the installation of WinGet onto a clean instance of either Windows Server or Windows 11 Sandbox, which I could then use to begin iterating on a Crescendo module.

Other than automating the installation of WinGet, the other major difficulty was with parsing the output of it's CLI. A list of packages could only be expressed in a columnar table format with both dynamic columns and widths, which meant the only way to parse the output (when it wasn't getting truncated by the CLI itself) was to calculate string lengths based on column headings with several different localization options.

Both the WinGet package provider and underlying Crescendo module can be installed via PowerShell Gallery.