where does this go on davepedu.com

John Henry in the age of AI

dave@davepedu.com (dave) — Thu, 06 Aug 2026 00:00:00 +0000

There is an American folk tale about a man named John Henry. It dates to the mid 19th century, a time during which America was rapidly industrializing and the use of steam power was picking up… Well, steam.

Being a folklore tale, there are many different variations of it, though, most examples describe him as a man employed on the construction of railroads. His specific role is often described as a “steel-driving” man but this takes various specific forms such as hammering steel drills into rock to dig railroad tunnels, or hammering the spikes that hold the rails to the wooden sleepers. Generally speaking, John Henry is depicted as an extremely skilled and well respected worker.

There are many full-text versions available online, but I’ll specifically reference this one. The synopsis of the tale of John Henry is as follows:

John Henry is a railway construction worker and sees a steam-powered machine performing his job at an incredible pace.
He decides to measure his prowess as a railway construction worker by having a duel against the steam-powered machine. The duel is usually something along the lines of racing to mine a railway tunnel or set and spike down a length of track, with the winner being the fastest to complete.
John Henry ultimately beats the machine, but the exertion he put his body through to do so causes it to give out, and he passes away.

It is the classical man-versus-machine tale. And I think the parallels that can be drawn between it and the emergence of AI in recent years are clear, and I will discuss them later on.

I heard the story of John Henry when I was a child, but I forgot all about it until just recently. I subscribe to several different interpretations of the folklore tale, but, in my opinion, it ultimately boils down to a statement on how ever-advancing technology disrupts and conflicts with human dignity, perseverance, ability, and purpose. It is a reasonable conclusion that John Henry felt threatened by the emergence of technology that could replace him. And his response of setting up a duel against said technology suggests that he wanted to prove that he, a human, can equal or defeat the machine. And the outcome of the duel asserts that man, the human spirit, ultimately triumphs over machine, but at a cost. This time, the cost was John Henry’s life.

Looking back on this story almost two centuries later, with a modern perspective over the technology we have now, coaxes out some obviosities. The use of machinery in tunneling and railway construction are the norm. Construction without machinery and with only human labor is practically non-existent anywhere machinery is available. There are still “steel-driving” men, but their tools are no longer hand tools like John Henry’s, but machines.

And the benefits of this shift to mechanization are obvious: such projects are completed faster, consuming fewer resources, and - most importantly - more safely and with a lower human cost. Other external factors like safety standards evolving over time too absolutely come into play here. However, having fewer humans on the job site, and more machines, ultimately expose less humans to harm for simple mathematical reasons.

I assert that the meaning behind the tale of John Henry is directly applicable to what is happening right now. It can easily be interpreted as an allegory describing the emergence of AI and the impact that AI’s emergence has on how labor is executed and what the value of a human being is with respect to work.

Let’s interpret the tale of John Henry in the context of the last couple of years of technological advancement and the emergence and adoption of - and opposition to - AI. I’m a software engineer, and will largely speak from the perspective of my career.

Software engineering is a field some speculate will be replaced by AI. Boris Cherny is the creator and head of Claude Code at Anthropic and, Anthropic is currently one of the top companies that actually creates AI, rather than consuming it. Cherny famously stated that “coding is largely solved” - to the chagrin of many software engineers everywhere.

That’s our first parallel to John Henry. A machine has been created that, in theory, can take the place of a human and do a task a human was previously required for. This corresponds to the steam-powered machine John Henry dueled against.

Sidebar: I completely understand that you can’t remove a human from their desk and plop an AI there and get 1:1 results. I am not making the argument that AI is sophisticated and accurate enough that this is doable. I write code for a living. I have seen the code that AI generates. They are not on the same level yet. Yet. I am also not commenting on whether I think AI will ever reach that point. It is inarguable that technology advances and gets better, but it is not a given that all technologies advance and get better. Plenty hit their limit and do not evolve further.

Let’s get back to our steam and silicon powered machines.¹ Watching the emergence of AI, I have absolutely felt the threat of it, personally. Watching a machine in seconds churn out a Python script that would have taken me at least an hour to write is frightening. Why pay me to spend an hour doing that when a machine can do it instantly? It felt like looking at my replacement. And I can imagine that John Henry, nearly two centuries ago, felt the same dread looking at the steam-powered machine doing his job.

How would my story look if I decided to duel the machine replacing me, like John Henry dueled his?

I’ve been programming since the mid 2000s, and professionally since 2010. I have always followed emerging technology, including AI, and looked for ways that I could leverage them to be a better developer. By now, I have a decent amount of experience using AI tools to execute my craft.

That being said, I look at the output of AI tools regularly and I have observed that it generally does not compare to code written by skilled humans. But that is not too surprising. Communicating the requirements of a software project often trips up humans too. There is an entire field of work dedicated to communicating and tracking requirements - project management. So it comes as no surprise that a bot lacking context outputs code that misses the mark. On top of that, AI is technologically limited by constraints like context windows that can easily lead to similar breakdowns.

But I also recognize that the code LLMs output is often “good enough”. It completes the task, but is written with patterns I don’t agree with or know better approaches to. And sometimes, that’s okay. The output can serve as a general skeleton, ready for improvements by humans. It can make assumptions that even a human missed, and now needs to consider. Or if the task is trivial enough, say one-off tasks, “good enough” might just be good enough.

I also need to mention that sometimes AI writes code exactly as I would have written it, which feels amazing and magical.

Quality does matter to me. I cannot stress enough how much it pains me to ship ugly, but working, code. Even outside the context of AI. I fully recognize Steve Jobs to be a deeply flawed man, but I adore the story of his interaction with his father in which his father explains the value of a job done right and with care.

The story of John Henry doesn’t touch on the quality of the work or whether it mattered to him. But, his position on the matter can be inferred. Versions of the tale include:

He picked up a hammer and a little piece of steel

Saying, “Hammer’s going to be the death of me, Lord, Lord,

Hammer’s going to be the death of me.”

John Henry dedicated his life to his craft.

And that brings us to the crux of what I’m trying to convey:

He cared about the craft and pride of the work itself, not just the outcome. And the same is true when it comes to AI: If you care about making and doing, and the process is more important than the result, AI isn’t for you.

If you’re trying to get a job done, AI can be a force multiplier. The code might not be great, but it just might work. Human code needs review and tweaking, and that doesn’t go away with AI. Human-written code can still be slop and that slop can get sent to production if it goes unreviewed.

These are the two extremes. The far, far ends of the spectrum of AI use. What defines the reasonable middle ground?

Quality defines the middle ground. A skilled developer can use AI as a force multiplier to write more code at the same quality he would have otherwise, but faster. And that’s what I think a major part of the debate around AI is. What I’ve revealed here is that it all boils down to what one’s viewpoint of the purpose of a human being is.

Everybody has to work to make the world and society go around. A lot of people think this sucks, and working your entire life is a tragedy. And I’m inclined to agree.

But I also see technology as an escape from this existence. In John Henry’s time, digging a tunnel required the toil of thousands of men and women. Or, similarly, farming required hundreds of men or women working the fields, becoming sunburnt and ruining their backs bending down to reach the plants on the ground. These occupations were dangerous and took an extreme toll on the body.

But those jobs are done differently now. Farming no longer requires thousands tending the fields year-round, almost all of the work can be done by machine, with one machine replacing many, many workers. Ditto for tunneling.

The men and women that would have been dying or destroying their bodies doing such labor are now doing something else. That something else might not take a similar toll on their body. And that’s a good thing. I’m describing a net reduction human suffering. And I’m describing manpower that has been freed up that can now be put towards something else that improves humanity, like figuring out how to replace jobs that are necessary but incur human suffering with solutions and machines that don’t suffer.

However, it’s worth mentioning that shift doesn’t happen overnight. People out of work because they were replaced by automation can have a hard time finding a new job. Starting over and shifting to a new field of work is often necessary due to the very automation making the job market in a specific field disappear. Some people become poorer. There are immediate and real world consequences worth considering. This is a legitimate grievance.

But, machines replacing humans isn’t inherently a bad thing. It can be beneficial as it frees up labor resources - those humans - and potentially allocates them to fields that advances the knowledge and capabilities and comfort of the human race.

And it’s a snowball effect. The observation that technology is advancing at a faster and faster pace is a common one, and accurate in my opinion.

I dream that, one day, machines are intelligent and accurate enough that mankind is freed from labor entirely. That one day we’ll hit a point of technological advancement that allows most, if not every, necessary function in society and the world to be handled by machines that don’t need to be paid and don’t suffer. And, unlike the above descriptions where this now-freed up manpower can be redirected into some other form of labor, it can now be directed into leisure. I feel that I need to mention that I believe some roles will probably still need a human touch forever. Social issues, perhaps.

The non-financial cost of AI is a hot topic and worth connecting to John Henry as well.

It has been said that the resources consumed by AI damage the planet. AI requires, among other things, electricity, and it is inarguable that many forms of electrical generation harm the planet. Burning coal, for example. And that’s exactly what powered the machine in the tale of John Henry.

We have the benefit of hindsight here. We can observe that over the past centuries that the burning of coal - releasing greenhouse gasses - has done measurable damage to the planet. Damage aside, descriptions of the absolute filth and constant depressingly grey skies of 19th century cities, such as London, boggle the modern mind.

But we can also see that old, harmful technologies such as coal are being replaced by better alternatives. Steam engines gave way to diesel, which perform significantly more work and output less pollution. Even diesel gave way to electric engines, which are even cleaner.

Technology advances. Improvements to the things we want to do - like writing code or laying railways - always come at a cost. There is no free lunch. But those costs decrease over time. We get the same amount of work done given less input. Just look at the advancement of the power efficiency of CPUs or density of hard drives, or the improvement in miles per gallon in cars or even the shift to electric vehicles.

AI has real use, and it’s not going away. It has a cost, and while the level of impact of that cost is hotly debated, it is inarguable that this cost is going to go down over time.

AI feels like a step towards the utopia I described - where all work is taken care of by machines, and humans are free to live as they please and pursue what makes them happy.

This seems like a time to touch on the subject of creativity. A common criticism of AI is that it does not create novel solutions, but is simply an information-laundering operation that - using software engineering for example - steals code from someone else.

And there’s a point to be made there. AI models are in fact created by feeding things created by humans into them. Many opinions, some moral, some legal, exist on the matter. I expect a moment of reckoning will come, like the way it did for Napster.

One version of the tale of John Henry concludes as follows:

Well, they carried John Henry down the tunnel

And they laid his body in the sand.

Now every woman riding on a C and O train

Says, “There lies my steel-driving man, Lord, Lord,

There lies my steel-driving man.”

In the context of the story, the memory of John Henry isn’t preserved in a monument or a plaque. It’s preserved in folklore. The memory of his deed is remembered by “every woman riding on a C and O train”.

The story’s own image of his legacy is happening on board the very kind of machine he dueled. The railroad won in the sense that it kept running, kept carrying passengers, kept being the dominant mode of transportation.

The craft John Henry dedicated his life to became a footnote of the past, but he wasn’t forgotten by the world. He was preserved, but as folklore riding inside the machine’s success. And it is worth pointing out that the very occupation John Henry was laboring in was itself a form of the advancement of technology: the proliferation of rail-based transportation.

John Henry won his duel, and it didn’t matter. The railroad kept running exactly as it was always going to. This part of the story gets lost when we tell it as a tale of triumph of man over machine. Individual victory and industry-level outcome were never the same contest, and John Henry seems to have believed that winning one settled something about the other.

There’s a version of that same mistake available to us right now. The engineer who refuses to touch AI on principle, be it for quality or moral or resource-related reasons, who wants to out-code the machine to prove that human craft still matters most… I understand that instinct, I’ve felt it myself, and there’s real dignity in it. But it’s a duel, and duels don’t move industries. Winning it, the way John Henry won his, doesn’t change anything.

That’s different from the person who lost their job to automation and is still finding their footing, or the person objecting because their work trained the very model now competing with them. Those aren’t duels for pride, those are real costs and real grievances, and I don’t think they get resolved just because I believe the long arc bends toward less work for everyone. They deserve consideration.

John Henry’s duel itself was a fight to prove something rather than a fight for something. And I think it only delays the utopic fantasy I described earlier. The fantasy, which is the same thing every generation before this one has fought for without knowing it: fewer hours breaking your back, more of them living. Every labor dispute in the story of mechanization was, underneath the immediate fight, an argument about who gets to keep the gains from the machine. My hope is that the endpoint of that argument, run out far enough, is a world with no labor left to dispute over. And so, for the first time, nothing left to exploit either.

None of that arrives on its own, though. The eight-hour work day didn’t show up because factories got generous. People, real human beings, organized and fought for a share of what the machine made possible.

If the same pattern holds here, the utopia I’m describing isn’t something AI hands us automatically just by getting more capable. It’s something we’ll still have to fight for, the same way every generation before us did.

I just don’t think the fight looks like a duel.

Writing this sentence made me chuckle, as most electricity generation involves creating steam and using it to spin a turbine, which makes even modern AI arguable steam-powered. ↩︎

modernizing this blog

dave@davepedu.com (dave) — Fri, 10 Jul 2026 00:00:00 +0000

My personal website has existed in some form since the late 2000s, and like many hobby projects, it has been rebuilt far more times than strictly necessary. My taste in tech stacks has changed over time, and each version reflects whatever I was experimenting with at the time.

These are some of thee iterations it has gone through over the years, and what is to come:

2011 - WordPress

Early 2012 - A single word to mark my existence on the Internet

Late 2012 - Cheesy wood paneling and filler text

Early 2013 - A fully custom Python blog engine

Late 2013 - Drupal 7 on a new .io domain

2014 - Design changes

2016 - More design changes

Aside - this list was actually a lot of fun to put together. I had forgotten entirely about some of these iterations.

Out of all of these, the Drupal 7 iteration lasted, by far, the longest. Which, itself, is amusing. Drupal is implemented in PHP, and there is a lot of writing on the internet about whether PHP is the greatest language or a terrible language. But either way, it is undeniable that PHP is like a cockroach that will not die.

But why Drupal, anyways? It bills itself as being for Enterprise, which usually means large and hulking codebases. And it is. When I set up this site using Drupal, I was working for the Rochester Institute of Technology, building the school’s own websites in Drupal. It was already familiar to me from this, so it was easy enough to use Drupal for my own purposes.

But, I eventually stopped working with Drupal and web development in general as I moved into other roles - which I wrote about here - and I didn’t keep up with Drupal or PHP’s evolution. Which, for Drupal, actually would have required some serious re-writes as the theme rendering layer was changed pretty drastically from version 7 to 8. But that’s probably more detail about Drupal than anyone needs.

PHP too, changed. My Drupal website was written to work with PHP 5, which hasn’t been supported for years. Server operating systems have long dropped support for it. Between it and Drupal’s age, it’s honestly a miracle the site never got hacked.

PHP evolved, Drupal evolved, became more complicated, and I did not keep up. So now it is time to build something new, and to keep it simple this time around.

This time, I’ve decided to use Hugo, which is a static website framework. In summary, you give it your content formatted in Markdown which it will organize and feed into a template, which then in turn outputs HTML.

This post, too, is a Markdown file that contains the text you see and some metadata at the top:

+++
title = "modernizing this blog"
date = "2026-03-14"
author = "Dave"
cover = "img/IMG_1053_cropped.jpg"
hideCoverInSingle = true
description = "This blog has been migrated to a static website, using Hugo!"
tags = ["Programming", "Meta"]
+++

I’ll lose a few things by switching to a simple static site. I could update the Drupal website from anywhere - just log in and write a new post, or save it as a draft. And the comment section - sorry, you’ll have to write to me directly now.

I’m also switching to my original davepedu.com domain, instead of dpedu.io. The .io domain was trendy at some point in the past, and I’ve come to think that a classic .com is better anyway. On top of that, it allows me to keep the old Drupal site online at dpedu.io while I manually extract content from Drupal and re-write it for Hugo. After a time, that domain will redirect here as well. The old site didn’t have terribly much content, but I’m also going through unpublished drafts and making them into real posts.

The new hosting stack will be similar, too. Before, I had a bare-metal dedicated server rented from OVH. I went with dedicated servers since at the time I was working on computer infrastructure. Another big complex thing I don’t want to take care of anymore, hah. I’ll be sticking with OVH, but I’ll be using a VPS instead. A VPS is a good middle ground - over something completely hands-off like static hosting on Cloudflare - because it still gives me a place to play sysadmin, which I enjoy. And everything configured on it will be installed with a simple form of configuration management - but more on that in a future post.

Of course, I couldn’t resist adding a bit of engineering fun.

I’ve written a Go-based application that acts like a toy version of AWS’s S3 static website hosting. It’s called s3static, and is meant to be used with S3-compatible storage systems such as Minio or RustFS. But more about that project in its own dedicated post.

Traefik will also be involved. It’s meant for Kubernetes but works great with a text config file too. It also automatically handles TLS certificates via Let’s Encrypt, for which I used an unusual and convoluted process, which is a whole other story.

Hopefully, keeping things simple will help this version stick around. Either way, rebuilding the site has been a fun excuse to revisit old memories, clean things up, and experiment with a few new tools.

davepedu.com chief technology officer

highly available redis with redis-sentinel-proxy

dave@davepedu.com (dave) — Thu, 05 Mar 2020 00:00:00 +0000

Redis is a widely used and extremely fast cache server. Redis supports high availability out-of-the-box, but requires a cluster-aware client library to accomplish it. In this project, I explore making cluster-unaware clients compatible with highly available Redis.

High availability

What is high availability? First, let’s cover a few basic concepts.

Computer hardware and software are not perfect. Software can crash due to bugs or other problems. Hardware can fail due to physical problems that prevent it from operating, like a short circuit.

But in today’s world, software services that break sometimes just aren’t good enough. We could talk about the flight computers on an aircraft - these absolutely cannot fail, or the consequences for those on board could be deadly. Your car has a computer-controlled airbag that will save your life in the event of a crash.

Less serious scenarios exist too. Retail giants like Amazon.com are almost always online.

First, let’s define availability. Availability describes how reachable and usable a piece of software or service is, in that all incoming requests receive a response. If you’re able to open Amazon.com in your web browser and buy something, then Amazon.com is said to be “available”: it is online and able to fulfill the purpose for which it was created.

High availability, as the name suggests, describes a higher level of availability. That is, a system that is designed to continue operating despite certain supporting component failures. Generally speaking, this means running the software - and supporting components - across many separate computer servers and often in different geographical locations.

High availability represents a challenge in the software industry. Creating a program that runs across many computers has challenges and caveats that are completely different from those of programs that run in one place.

The CAP theorem is a brief description of the type of challenge faced. It was written to describe distributed data stores, but the idea applies to highly available systems in general:

The CAP theorem … states that any distributed data store can provide at most two of the following three guarantees:

Consistency - every read receives the most recent write or an error. Consistency means that all clients see the same data at the same time, no matter which node they connect to.

Availability - every request received by a non-failing node in the system must result in a response, without the guarantee that it contains the most recent version of the data.

Partition tolerance - the system continues to operate despite an arbitrary number of messages being dropped (or delayed) by the network between nodes.

- Wikipedia

In practice, network partitions are unavoidable, and systems must choose between consistency and availability.

But why does this matter? I’ll illustrate using an example:

Credit cards are a staple of modern society. In America, almost a third of all transactions are made using a credit card, according to the Federal Reserve. Thus, any disruption to the ability of consumers to use a credit card creates massive disruptions in everyday life, perhaps even to the point of being life-threatening. When issues like this happen, it makes headline news.

That makes it clear why availability is important: people have things to do, and they need your systems to do it.

But what about consistency?

Credit cards have a limit. When you make a transaction, the amount transacted counts towards your limit and the credit card provider may refuse the transaction if it would put you over the limit. This exists to protect you from accidentally spending too much money, and protects the credit card companies from lending you more money than they want to.

But what if you make two credit card transactions at the exact same time? Your credit card provider needs to be able to handle this situation. One transaction needs to be processed before the next, and the subsequent transaction needs to be processed using up-to-date data. For example, if your credit limit is $100 and you make two $75 transactions at once, one of them must not go through as it would leave you over your limit.

That all sounds rather obvious. But what about partition tolerance?

As mentioned above, highly available software is often run in different geographic locations. This is to protect against environmental threats to availability, such as natural disasters knocking down power lines or communications equipment. This stuff happens, and even if some servers are knocked offline, others exist elsewhere to handle the workload.

But, machines being located in separate locations create a risk for partitions: that is, when the two sets of machines cannot communicate with each other, but can still reach the outside world. To use the credit card transaction example again - for this example to work correctly, the entire fleet of transaction-handling servers needs a consistent view on how much money is in your account. This is impossible if they cannot communicate.

Redis

While high availability can be used to describe entire services, it can also describe subcomponents of a system.

Redis was described as a cache server above, but technically, it is a database server. And, as software systems are often backed by a database, it is often very important within a system. A system that needs Redis to function can only be as reliable as the Redis instances it communicates with.

As such, Redis has built-in support for high availability. The exact details can be read in Redis’s documentation, but the gist is as follows: a program called Sentinel monitors multiple distinct Redis instances, and orchestrates the instances so that if one fails, another instantly takes over.

This works great in practice, but using such a cluster requires some additional work. Specifically, Redis provides mechanisms and client libraries that are cluster-aware that integrates with Sentinel, so that from your application’s perspective, any turmoil in the Redis cluster is hidden away.

But not all software requiring Redis is written using this special library.

And that’s where redis-sentinel-proxy comes in.

Redis-sentinel-proxy

Redis-sentinel-proxy is a reverse proxy that exposes a standard - that is, non-cluster/Sentinel-aware - Redis connection, that standard Redis clients can use. My project is a fork of @enriclluelles’s original, and makes some reliability and usability improvements.

It works as follows:

Continuously query Sentinel for the Redis cluster’s status.
When a Redis client connects, connect the client to the active Redis instance.
When the Redis instance fails, close connections to clients and allow them to connect anew.

That’s it! Now, cluster-unaware Redis clients can take advantage of highly available Redis.

This was a fun project. If you find some use in it, let me know!

Source code: https://git.davepedu.com/dave/redisproxy

ubuntu kickstart / preseed builder

dave@davepedu.com (dave) — Sun, 07 Apr 2019 00:00:00 +0000

Kickstart and Preseed are tools for automatically installing Linux distributions such as Ubuntu. Installing new copies of Linux can be time-consuming, especially if you must do so from a ISO image. These tools automate the install process.

From the Ubuntu documentation:

Many system administrators would prefer to use an automated installation method to install Ubuntu on their machines. The preseeding feature of debian-installer answers this need.

Using Kickstart, a system administrator can create a single file containing the answers to all the questions that would normally be asked during a typical Ubuntu installation.

After creating a few ISO images with embedded Kickstart/Preseed configurations, I decided I needed faster iteration, and so I created a tool for generating Ubuntu Linux .iso files with embedded automated installers: Ubuntu Kickstart Builder.

The tool works in a fairly straightforward way:

Unpack Ubuntu ISO images in the tool’s data directory.
Open the web UI and edit the Kickstart and Preseed configuration files.
Optionally, upload additional files to be included in the ISO image.
Click Build, and your customized CD image will be downloaded.

The ISO image can then be written to physical media or booted in a VM.

While Kickstart and Preseed configurations can also be served over the network, embedding them directly in an installation image has some advantages. A self-contained ISO is easy to distribute, works in environments without network infrastructure, and guarantees that the installer always uses the intended configuration. This can be especially convenient for testing in virtual machines, provisioning isolated lab machines, or sharing reproducible install media.

pyods: open directory scraper

dave@davepedu.com (dave) — Thu, 28 Feb 2019 00:00:00 +0000

I’m a long-time reader and participant of the Reddit.com Subreddits /r/Datahoarder and /r/OpenDirectories. As both of these deal with retrieval and archival of data and information, wget is a wildly popular tool within these communities.

For the unfamiliar, an “open directory” is a directory on a website where there is no index file. Since there’s no content to serve, most web servers will generate a page showing a listing of files on the server. Wget is a command-line tool for making various types of HTTP and FTP requests. Specifically, wget is specialized for content mirroring and it works nicely with open directories. That is, following links at the target URL to discover additional links to download with the end goal of downloading an identical mirror of the remote site.

For the most part, wget excels at doing this. It’s been around for decades and is supported on nearly every operating system, is easy to find information about online, and is generally bug free.

However, wget suffers from a design flaw - its single threaded operation.

Meet Pyods - the modern, Python, Open Directory, Scraper:

$ pyods --help
usage: pyods [-h] [-u URL] [-o OUTPUT_DIR] [-p PARALLEL] [-c] [-d DELAY]
[-e EXCLUDE [EXCLUDE ...]] [-f EXCLUDE_FROM] [-v]
Open directory scraper
optional arguments:
-h, --help show this help message and exit
-u URL, --url URL url to scrape
-o OUTPUT_DIR, --output-dir OUTPUT_DIR
destination for downloaded files
-p PARALLEL, --parallel PARALLEL
number of downloads to execute in parallel
-c, --clobber clobber existing files instead of resuming
-d DELAY, --delay DELAY
delay between requests
-e EXCLUDE [EXCLUDE ...], --exclude EXCLUDE [EXCLUDE ...]
exclude patterns
-f EXCLUDE_FROM, --exclude-from EXCLUDE_FROM
exclude patterns from file
-v, --verbose enable info logging

Pyods was written, in Python 3, from the ground up with speed and mirroring in mind. In contrast with wget, Pyods downloads many URLs in parallel. The options in the help text above show off Pyods’ other capabilities such as excluding URLs, adding delay between requests, etc.

An example invocation might look like:

pyods -u http://example.com/some_open_directory/ -o ./output -p 10 --delay 5

This example would use 10 parallel downloads with a delay of 5 seconds between each HTTP requests.

I intend to release this on Pypi eventually. But for now, Pyods is available on my git instance. Happy scraping!

Source code: https://git.davepedu.com/dave/pyods

nginx upload size behavior

dave@davepedu.com (dave) — Fri, 14 Dec 2018 00:00:00 +0000

I recently encountered some odd behavior in nginx involving file upload sizes. Nginx has long had a client_max_body_size setting which controls how large of an HTTP POST or file upload the server will accept by comparing to the Content-Length header. Attempting to send a request larger than this results in a 413 Request Entity Too Large error from the server. But is this always the case?

For starters, here’s the description of client_max_body_size. From nginx’s docs:

Sets the maximum allowed size of the client request body, specified in the “Content-Length” request header field. If the size in a request exceeds the configured value, the 413 (Request Entity Too Large) error is returned to the client. Please be aware that browsers cannot correctly display this error. Setting size to 0 disables checking of client request body size.

While this sounds pretty obvious, there’s some hidden behavior one might not expect.

Take the following server configuration block for example:

server {
server_name subdomain.somehostname.com listen 443 ssl;
ssl_certificate "/some/file.crt";
ssl_certificate_key "/some/other/file.key";
ssl_protocols ...;
ssl_ciphers ...;
client_max_body_size 64m;
location / {
return 307 https://anothersubdomain.somehostname.com$request_uri;
}
}

The function of this configuration is obvious - it’s an ssl server that will issue a 307 redirect to all requests, redirecting them to another location. In my case this location would receive file uploads from legacy software; the 307 was needed to cause clients to retry the upload elsewhere. Of course, if the upload is larger than 64m, a 413 error is returned instead.

But what happens if we remove the location block?

server {
server_name subdomain.somehostname.com listen 443 ssl;
ssl_certificate "/some/file.crt";
ssl_certificate_key "/some/other/file.key";
ssl_protocols ...;
ssl_ciphers ...;
client_max_body_size 64m;
return 307 https://anothersubdomain.somehostname.com$request_uri;
}

While this looks like it would function identically (and it does for the most part), there’s a subtle difference. Nginx will no longer enforce the client_max_body_size directive! Requests containing uploads of any size will be issued a redirect.

This is of course a relatively minor issue, as a redirecting server does not process any upload content. In other words, the client is redirected before the upload process begins. In my case, I was moving from the latter config above to the former due to needing some additional location blocks in the config.

As it turns out, this behavior is likely undocumented. Lesson learned - be careful how you set a client_max_body_size!

isle of man TT

dave@davepedu.com (dave) — Thu, 14 Jun 2018 00:00:00 +0000

Earlier this year, I traveled from my home in California to the Isle of Man, which is a 35 mile long island in the Irish Sea, known for its motorsports events. The Isle of Man is a self-governing British Crown Dependency, meaning they are effectively their own country, though they technically belong to the British monarchy.

The Isle of Man is most well known for the Isle Of Man Tourist Trophy - a 2-week motorcycle racing event held on the island’s public roads. For those two weeks, the public roads close down on weekdays, and are used instead as a roughly 38 mile long circuit raceway. In addition, many public events are held during the race weeks. It’s quite the spectacle.

The options for lodging on the island are actually pretty interesting - the local government runs a program called Homestay, which encourages residents to rent out extra bedrooms to tourists. It’s a pretty great experience - much more interesting than a hotel, and the rates can be lower too. This is what I did - I rented a bedroom in the town of Peel, the third-largest town on the island. More on Peel below.

This album covers sights from the island and along the way, so there is something for everyone.

A custom motorcycle parked on the streets of Douglas.

I live in California, and the Isle of Man is about 5,000 miles away from me. Its airport is small so there is no direct flight from here, so between my various connecting flights I traveled about 6,000 miles in total - one way. Fun fact - the “halfway around the world” idiom if taken literally is about 12,000 miles, and I traveled half of that. Halfway to halfway around the world!

A layover of about a day at London Gatwick Airport allowed me to spend the day in London. I took the classic double-decker bus tour, and a boat tour on the River Thames. I didn’t have a set agenda besides these items, and effectively wandered otherwise. I first found myself in Hyde Park, which features many historical monuments.

Australian War Memorial

From Wikipedia:

The Australian War Memorial in London is a memorial dedicated in 2003 to the 102,000 Australian dead of the First and Second World Wars.

New Zealand War Memorial

From Wikipedia:

The New Zealand War Memorial in London is a memorial to the war dead of New Zealand in the First and Second World Wars, unveiled in 2006. Officially named “Southern Stand”, the memorial was designed by architect John Hardwick-Smith and sculptor Paul Dibble, both from New Zealand.

Big Ben

Big Ben was undergoing some sort of construction work that required it to be nearly enclosed in scaffolding. Unfortunately, you couldn’t see most of it. The Palace of Westminster is on the left.

Close up of the clock face, behind which Big Ben - which is technically the name for the bell inside - sits.

Tower Bridge

We were going to cross it on the bus earlier, but the driver informed us we could not because some sort of protest was taking place on the bridge, blocking traffic. This is seen from the dock where the boat tours launched from.

classic London black cab

London Victoria Station

Having spent the morning and early afternoon exploring London, the time came to catch my flight to the Isle of Man. I took the Gatwick Express out of London back to Gatwick airport.

race day

This isn’t a bike competing in the day’s races, but it was race day on the island! The number of motorcycles that appeared on the island was impressive. Fields upon fields of parked bikes.

Douglas

The race was on! This is a racer speeding through Douglas, the largest town and capital of the Isle of Man. You can see that they literally do just use the island’s public streets for the race course - and the public comes out to watch.

It’s difficult to describe how fast these racers move. It’s nothing like standing next to a highway - these bikes move at more than double highway speed - on average! The record average speed for an entire lap is 136 MPH, held by Peter Hickman. Of course, this average speed includes slower parts of the course. On simpler portions of the course, such as straightaways, speeds over 200 MPH are frequently attained.

sidecar race

In addition to motorcycles, races with “sidecar” motorcycles are also run.

Really, they aren’t motorcycles under the hood at all. It’s not a motorcycle with a sidecar added. It’s a completely custom machine for this specific class of race.

During the race week, races happen Monday / Wednesday / Friday, with Tuesday and Thursday being break/prep days. This downtime in between race days gave me plenty of time to look at other sights on the Isle of Man.

cat

Cat.

Douglas Promenade

The island’s capital and largest city, Douglas, is situated on a small bay, and features a seafront walkway known as the Promenade. During TT week, many events and shops set up here. The race doesn’t come though here, but this can be considered the city’s downtown.

Tower of Refuge

The Tower of Refuge is a short distance out to sea in Douglas Bay, and can be seen from the Promenade. It was built as shelter for shipwrecked sailors. The coastline along Douglas forms a nice bay, but this island, St Mary’s Isle aka Conister Rock, is part of a submerged reef that posed significant risk to ships historically. So, in 1832, the Tower of Refuge was built as a miniature castle as a shelter for shipwrecked sailors.

Bench along the Promenade, displaying the Isle of Man’s National symbol, a triskelion.

Suzuki V-Strom

Seen parked along the Promenade. This is the same model of bike I own back home. I don’t see many of them in the wild, probably less than 10 instances since I’ve had it.

Town of Peel, evening

I had rented a room in the town of Peel as mentioned earlier, so I returned here once it became evening. During TT week, the island runs many and frequent bus routes, which made getting around the island really easy, and relatively inexpensive. Peel is on the opposite side of the island compared to Douglas, but it wasn’t a very long bus ride - about 10 miles.

I’m told that the word “peel” in the Manx language - a Gaelic Celtic language spoken by the island’s original inhabitants - means “castle”, and as such, Peel’s bay and coastline is dominated by the ruins of a castle, to the right side of the photo.

same spot overlooking Peel Beach, next morning

Peel Beach

Eurasian Oystercatchers on Peel’s beach.

Peel Lighthouse

A small lighthouse on the breakwater that protects Peel’s marina.

Peel Castle

Seen from Peel Beach. Nice sunlight this morning.

A wider shot.

Looking into the castle’s courtyards from a nearby hill.

Inside the castle.

Further inside the castle.

Another triskelion flying above the castle.

Coast Guard station

The island’s Coast Guard operates a station on Peel beach. Here, they have a lifeboat to assist ships in distress. On a moment’s notice, the boat is moved out of this enclosure and down a ramp into the sea. The tractor in the foreground is a Talus MB-H amphibious tractor.

Peel, from above

There is a large hill on the west side of Peel that gives a great vantage point to see the town from above. Looking northeast.

hiking from Peel

There is a hiking trail that encircles the island, known as the Coastal Path. I walked a portion of it departing from Peel. A large percentage of the island’s coastline is impassable rocky cliffs like this, which makes the land of little use beyond farming.

livestock gate

While the hiking trail is public, it passes through many different farms, each separated by short rock walls. These peculiar gates are installed along the wall to allow hikers to pass though. The gate is designed with an unlatched and movable swinging portion in the center. It’s shaped such that a livestock animals - sheep and goats were common - cannot pass through the gate, but humans can.

Ramsey, another race day

The next day, I found my way to the town of Ramsey, near the north end of the island. My impression of Ramsey is that it is more of a blue-collar working town, particularly around the fishing industry.

Conrod’s

I saw this cute and on-theme coffee shop, sign complete with engine parts and another triskelion.

action!

Ramsey was a great place to watch from, featuring several straightaways and hairpin corners.

One particular corner had a pub’s outdoor seating area situated in it. It was a great place to watch the bikers whiz by, mere inches away, while you were being served food and drink.

Isle of Man Motor Museum

Located relatively close to Ramsey is the Isle of Man Motor Museum. Its indoor showroom features countless rare - but also some common - cars, trucks, motorcycles, military equipment, and other vehicles. Active restoration work was going on within the museum’s workshops as well.

The Peel P50 - as seen on Top Gear - claimed the title of being the smallest production car in the world could also be seen here.

under the Queen’s Pier

The Queen’s Pier in Ramsey was built in the 1880s as a place for ships to pick up and drop off passengers. It was closed in the 1990s due to safety concerns and remained closed for about 25 years. At the time of my visit, restoration work was just getting started, and the pier has since reopened.

Towards the bottom of the photo, you can see how rusted out the metal supports are.

Another view of the Queen’s Pier, from a distance so that we may see its length.

electric rail

The island runs a historical electric railway, called the Manx Electric Railway. Again dating from the late 19th century, the railway runs on 550 volts DC and features roughly 25 electric trams. It holds the title of being the the oldest electric tram line in the world in the sense that most of its electric trams are original, some dating back over a century.

A close-up of the hand-painted insignia on the side of the electric tram.

The railway snakes along the eastern coast, providing excellent scenic views of some quieter parts of the island.

Laxey and Laxey Wheel

The electric rail eventually brought me to the village of Laxey, where there is a small station and junction. From here, one can take another train up Snaefell Mountain, the highest peak on the island, or view some attractions in Laxey. Pictured above is Laxey Wheel, and enormous waterwheel. It holds the title of being the largest surviving original working waterwheel in the world, built in 1854.

Laxey is a historic mining town. In the 19th and early 20th century, large amounts of lead and zinc were mined here. The ore was washed on various washing floors - basically wide open spaces with flowing water - that still exist today.

Laxey Wheel served as a water pump to deliver water to these floors and other mining operations. The wheel itself is about 72 feet in diameter, 6 feet wide, and spins at 3 rotations per minute.

The wheel is now a National Heritage site.

Snaefell Mountain Summit

From Laxey, I took the electric rail up to the summit of Snaefell Mountain. There is a small restaurant at the top, but it seemed to have been closed for some time. You could go inside in case of bad weather, but no food was available.

The railway takes you right to the top. No hiking required!

I’ll be honest - taking a century-old vehicle up a mountain did seem a little scary. The railway was steep and wound around the contours of the mountain, making working brakes absolutely critical. Looking back after some research, the railway does have an excellent safety record, which puts my mind at ease.

An electric tram, departing.

Scenic view from the summit.

Calf of Man

Later, I made my way down to the southernmost tip of the main island. Not far from shore is another island, known as the Calf of Man. The name comes from the island’s shape; it roughly looks like a lamb calf - distinct legs, a body, and a head section.

The Calf of Man is uninhabited, but there are a few historic structures on it.

Sea wildlife, near the Calf of Man, in an area known as Cow Harbour.

hiking to Port of Erin

From the shorelines near the Calf of Man, hiking trails depart allowing you to walk either towards the east side of the island, or the west side. I chose the west side, taking a trail that would eventually arrive at the Port of Erin.

Despite being only about 3 miles long, I found the hike somewhat difficult. As you can see in the image above, the coastline the trail follows slopes steeply towards the ocean. This made walking difficult as your ankles constantly wanted to roll sideways down the hillside.

Wildlife along the trail.

Trail signage. “Raad Ny Foillan” - Manx for “The Way of the Gull” is the same trail seen earlier near Peel that encircles the island. I believe this sign is marking the fork in the trail that takes you either on the trails along the west or east side of the island. The coastline of the island meanders enough that despite the entire island being only about 30 miles long, the trail’s length is nearly 100 miles.

Milner’s Tower

When approaching the Port of Erin, a picturesque stone tower can be seen on a hill overlooking the port’s bay. Milner’s Tower - built atop Bradda Head - was built to honor a local businessman, William Milner, in the mid 19th century, for his love and support for the town.

Closer view of Milner’s Tower tower. It’s tall towers and narrow footprint gives it an extremely fairytale-like appearance.

Port of Erin beach

Port of Erin is an adorable little town, seemingly a hot spot for beach-going tourists.

Isle of Man Railway

Separate from the electric railway in the north of the island, the southern half of the island is serviced by a historic steam railway. Today, the trains run from Port of Erin to Douglas. I took this train from Port of Erin, stopped in Castletown along the way, and eventually made it back to Douglas where I called it a day.

final race day

The next day was the final day of racing. On the final day, the main event - The Senior TT - is run. This is the event the island is known for.

Pictured here is a bike and crew participating in the race. Just like in other motorsports, the crews keep heaters on the bike’s tires right up until go time, as seen here. Hot tires grip the roads better, allowing racers to corner faster and make better time.

the end

This concludes my time on the Isle of Man. I stayed on the island for almost two weeks and had time to see many of the historic and cultural sites on the island, in addition to the motorsport races. The Tourist Trophy races have been run for over a century now, deeply intertwining the two. Learning about the island’s history and culture was horizon broadening. As someone who grew up in a small town, the layout and scale of the Isle of Man made me feel right at home.

I was very fortunate to have been visiting at a time when some of the best weather in years was rolling though. The British Isles are famously rainy and dreary, and the Isle of Man is no exception. However, we had clear weather nearly every single day during my stay. I recall about an hour of rain during one cloudy day, but no more.

Here, on the northernmost point of the island, stands a lighthouse called The Winkie. That’s me standing on it!

the author

The author of this blog, photographed at Jurby Junk, a sprawling “bric-a-brac” store.

I hope you enjoyed reading about my trip. I sure enjoyed the trip itself, and writing this post. The race was amazing to see, and staying with the locals rather than sleeping in a hotel made the trip a truly special experience I’ll never forget.

If you enjoyed my story and photos, or are planning a trip of your own, let me know!

shifting sands

dave@davepedu.com (dave) — Tue, 24 Apr 2018 00:00:00 +0000

I’ve ran this blog in its current form for about 5 years now. It runs on Drupal 7, which while still current and supported, dated in quite a few ways. Drupal has since released Drupal 8, which includes a massive overhaul to the internals and theme layer. I’m undecided as to if I want to upgrade the site to Drupal 8, as it would effectively require a re-structuring and re-write.

The various “Drupalgeddon” vulnerabilities have made it clear Drupal has holes and people are looking for them.

I guess we’ll see what happens.

old code

dave@davepedu.com (dave) — Thu, 26 Oct 2017 00:00:00 +0000

I recently came across the source code of a browser-based game I wrote around 2005, when I would have been 15 years old. While the original database structure was unfortunately lost, I managed to piece together a working version by looking at existing queries in the source code. With this, I was able to get the game running again! In this post, we’ll look at what a naive PHP developer can create - and the mistakes they make.

why the web

I can’t remember the original reason why I wanted to make a website, or how I got into web programming at all. I recall taking an HTML class in high school, but by that time, I was already proficient in HTML and even writing PHP, CSS, and some JavaScript.

I was able to retrieve, via archive.org and other methods, some old examples of web sites I made around this time. And, judging by them, I apparently wanted to make websites about forts in the woods that my friends and I built when we were kids. No pictures, because digital cameras were uncommon and expensive back then, but some had MS Paint style maps drawn of what we’ve built and where it is in the wilderness.

I do recall another force that brought me into PHP and web programming: online MMORPGs. Like many young teens at the time, I was addicted to the online game RuneScape. And I also recall setting up forums for the group of players I hung out with online - taking an open-source PHP forum - phpBB - and installing it on a free php/mysql web host. I recall installing mods - source code mods to add features - into the forum software. This wasn’t automated with plugins like modern software. The “mods” were source code patches. Often, if you had multiple mods, the diffs would conflict and you’d have to resolve differences manually. I think this is where I first started looking at PHP.

Beyond that, it was “cool” at the time on online forums to have a dynamic signature. Just like a signature in an email client, a signature is a message that gets attached to the bottom of every post you make. You could embed images, so it was possible to hotlink to an image that gets generated on the fly.

But back to RuneScape forums. The most common kind of dynamic signature was one that shows the level of your character in the video game. Players put work into games and feel proud of the achievements they make, and this is especially true about RuneScape as it is notorious for its grueling time commitments. It is not surprising that such a thing became a popular trend. Players want to show off their work, and a flashy image that does so under every post they make is a great way to do it.

Young me thought that was cool too, and young me thought it was even cooler if I could have a completely unique dynamic signature, unlike anyone else’s. It would be showing off my programming achievement in addition to whatever the signature itself displayed.

And I did. And it worked. And I learned about things like caching, even though I didn’t even know the word for it at the time. I found through trial and error that making my dynamic signature script call cURL and fetch information from RuneScape’s website was too slow. Even back then, they would automatically throttle and block scrapers like mine. And so caching became a necessity.

The addiction grew from there. Both the game addiction and the programming addiction. The rest is history, as they say.

Beyond RuneScape, I was also a fan of top-down monster-fighting RPGs such as the Dragon Quest/Warrior series. I recall making other games in the past with ready-made game making software such as Kyle’s Quest, but never anything involving code or starting from scratch. RuneScape’s 3D graphics were far too much for me to match, so I apparently chose a 2D top-down style comparable to these other games.

the game

The UI is simple - buttons to move in 4 directions. A button to wait which refreshes the page. Page loads were a little slower in the 2000s, so I had a walking queue that used a little JavaScript to send many actions at once. I guess I wasn’t the greatest speller then - “que” not “queue”, heh. On the battle screen, options to attack and run. That’s it!

The game is technically an MMORPG: you see other players moving around the map with their name displayed over their heads. There are level leader boards to drive competition between players. Being a web game, being turn-based made the most sense. One page load equals one turn. However, the game had so few actual features - no chat, no inventory - it’s hard to put it in a category at all, besides that of being just a fun programming challenge for young me at the time.

I was never much of an artist, so the graphics were stolen from other games. I believe the trees and buildings and general scenery came from the Nintendo Entertainment System version of EarthBound. The player character itself came from A Day At Work for Mac OS. Other graphics, such as enemy monsters, were arbitrary images, such as the monster.com monster. The website itself that housed the game was from a now long defunct free template website.

The gameplay loop was simple. The starting area was safe. Go to a dangerous area by going the only way you can go, into the house and out the back door. Wander around until you encounter a monster. Fight it, die, and respawn in the starting area. Or win, and gain experience points. Wander and fight again, or go back to the starting area to heal. That’s it!

a closer look

The game is overly simple, I have no disillusion about that. It is, however, technically a game engine. Content is decoupled from the engine. I can’t recall if I understood this concept at the time. Probably not. Or perhaps, though experimenting with RuneScape private servers, I had observed such a decoupling of content and engine and internalized it without even knowing the name of the technique. I’ll never know.

Just because it’s a game engine, doesn’t mean it’s a good game engine. I’ll talk about the funny parts first.

Being a mid 2000s PHP app written by a teenager, it is obviously full of security issues and other uncleanliness that would not surprise an experienced programmer.

Apparently I didn’t know about PHP sessions data yet, because I used raw cookies to handle user login. And by raw, I mean raw:

if($_POST['u']!="") {
$u=addslashes($_POST['u']);
$p=addslashes($_POST['p']);
if($u!="" && $p!="") {
$sql="SELECT * FROM mrpg_users WHERE name='$u'";
$result = mysql_query($sql);
while($row = mysql_fetch_array($result)) {
if($row['password']==$p) {
$message="You have been logged in. Proceed to the Game.
";
setcookie("mrpg-ethbnd", strtolower($u . "@seperator@" . $p), time()+525600);
} else {
$message="Invalid username or password.";
}
}
}
}

The cookie name is hard-coded on line 10. It’s also hard-coded in every file. I knew about includes, there was one a few lines up in this snippet’s context, but I will never know why I didn’t use one for something like this.

But that’s only the beginning. Perhaps you noticed what the cookie is set to on line 10 as well. It’s the literal username and password separated by a hard-coded string, @seperator@. Very strange, and also reveals that raw passwords were being stored in plain text. Terrible practice, but it did work. At least I understood the separator token had to be something players were unlikely to type.

I cannot recall why I chose 525600 seconds as the cookie expiration time, either. 525,600 is the number of minutes in a 365 day year, so I think a reasonable guess is that I wanted a year and thought the duration was in minutes. But as written, the cookie expiration ends up being slightly longer than 6 days.

How this cookie was read isn’t much better. In every game file, the same snippet was repeated again:

$credsYo=$_COOKIE['mrpg-ethbnd'];
$credsYo=explode("@seperator@", $credsYo);
$u=strtolower($credsYo[0]);
$p=strtolower($credsYo[1]);
if($u=="") {
header("Location: index.php?login");
}
$result = mysql_query("SELECT * FROM mrpg_users WHERE name='$u'");
while($row = mysql_fetch_array($result)) {
if($row['password']!=$p) {
header("Location: index.php");
exit;
}
}

So many problems in so little code. The same hard-coded cookie name on line 1. With the same hard-coded separator token on line 2. We only support lowercase passwords, apparently, because of the strtolower() on line 5. Attempting to read index 1 from the cookie credentials probably causes an error if the cookie isn’t set at all. It looks like we redirect the browser on line 8 with the header() call, but the script keeps running because there is no exit;. If there’s no cookie, we look up a blank user with a blank password on line 11. We check the plaintext password vs the database plaintext password on line 13.

But if all of this passes, we have actually verified which user is logged in. Barely.

This is probably also vulnerable to SQL injection via the username as there is no attempt made to sanitize $u the username before inserting it directly into a SQL query. This vulnerable snippet appears everywhere!

Many other places in the code rely on addslashes() for SQL injection protection. While somewhat sufficient for plain ASCII, it is likely this does not block SQL injection as this sanitation method isn’t fully sufficient. It can fail due to text encoding misalignment and is ignorant of the SQL server’s supported escaping modes.

These are the most egregious problems. Other lesser issues are more naiveties than actual problems

The way the “walkable” status of the map is done is amusing:

$walkable[56]=array(0,0,0,0,0,0,1,0,0,0,0,0,0,0,0);
$walkable[57]=array(0,0,0,0,1,1,1,1,1,0,0,0,0,0,0);
$walkable[58]=array(0,0,0,1,1,1,1,1,1,1,0,0,0,0,0);
$walkable[59]=array(0,0,1,1,1,1,1,1,1,1,1,1,0,0,0);
$walkable[60]=array(0,0,0,1,1,1,1,1,1,1,1,1,0,0,0);
$walkable[61]=array(0,1,1,1,1,1,1,1,1,1,1,1,1,0,0);
$walkable[62]=array(0,0,0,0,0,0,0,0,0,0,0,0,0,0,0);

This is an excerpt, but the above linked file has much more content just like this. A wall of textg. And what this wall is, is a giant array of 1s and 0s that specify whether each coordinate in the game can be walked into. The array’s indexes are the world map’s coordinates.

This isn’t too bad, actually. Probably fairly efficient, if it were to grow. And it’s simple.

However, one of my pet peeves shows up in this file too:

if($walkable[$y][$x]==1) {
return true;
} else {
return false;
}

This should be simply return $walkable[$y][$x]==1;! No need to if and else! Just return the boolean! Argh!

When rendering the map, the game needs to fetch a list of the other players so it may know where to draw them on the map. The code that does this simply fetches all users!

$result = mysql_query("SELECT name,x,y,lastact,dir FROM mrpg_users");

If this game ever saw a serious amount of players - which it did not - this would be slow as many many players images would need to be rendered, even those invisible off-screen. And as more players joined, the map would render slower and slower. A location filter should be used here, that excludes players too far away to be worth rendering.

The map render script was otherwise somewhat slick. It utilized PHP’s GD library, which is a set of APIs for manipulating images. You can load images, combine them, write text, crop, scale, etc. Effectively, my game’s map rendering process is that it queried the DB for players, calculated the map pixel position based on the current player, and drew each player on the map. Pretty simple.

Other screens, such as battling monsters, had separate render scripts that retrieved and displayed information in the same manner. Image-based instead of web and text based is an odd choice, but I guess my early experience with dynamic forum signatures had me liking the GD library.

Lastly, digging through the source code also revealed that each character had a wide variety of different properties …most of which never actually got used for anything:

imageString($im, 4, 50, 100, "PP: " . $row['pp'], $white);
imageString($im, 4, 50, 140, "Offense: " . $row['offense'], $white);
imageString($im, 4, 50, 180, "Defense: " . $row['defense'], $white);
imageString($im, 4, 50, 220, "Fight: " . $row['fight'], $white);
imageString($im, 4, 340, 100, "Speed: " . $row['speed'], $white);
imageString($im, 4, 340, 140, "Wisdom: " . $row['wisdom'], $white);
imageString($im, 4, 340, 180, "Strength: " . $row['strength'], $white);
imageString($im, 4, 340, 220, "Force: " . $row['force'], $white);
imageString($im, 5, 200, 260, "Level: " . level(2, $exp['level']), $white);
imageString($im, 5, 200, 275, "EXP: " . $exp['level'], $white);
imageString($im, 5, 190, 10, "HP: " . $row['hp'] . "/" . $row['maxhp'], $green);

HP, EXP, Offense and Strength were used, but the others did nothing. All of them increase when leveling up, but still are never read.

epilogue

I hope I did an understandable job of explaining how the internals of my naive game work, and that the problems called out that I wrote into it aren’t too offensive - remember, I was 15!

I wrote a Docker Compose file should you want to run a copy of the game. In addition to the code vulnerabilities - which I will not be fixing because they add charm! - the Docker images used are fairly outdated as well. So, it would be a bad idea to put this on the public internet. But if you want to play, it’s available.

Source code is available on my Git instance:

https://git.davepedu.com/dave/moprpg

Think you achieved a high score in my game? Let me know directly.

single system image linux clustering

dave@davepedu.com (dave) — Thu, 10 Aug 2017 00:00:00 +0000

Recently, I came across the concept of SSI - “Single System Image” - Linux clustering. In short, Single System Image is an approach to clustering where many generic Linux computers, running a modified kernel, present themselves as one larger computer with access to the combined sum of all the underlying systems’ RAM, CPU cores, and other resources.

It was a useful technique because it decoupled your workloads from your hardware, which is the same goal as today’s cloud computing platforms. Your hardware can be generic, you just need enough total silicon for whatever your workload is, and this technology makes it all work together.

This approach, along with some other nifty Linux boot tricks, allows for some interesting ways to scale. These ideas are mostly superseded today by modern cloud computing or solutions like Kubernetes, but I think they are interesting enough to be worth discussing.

What I will attempt to describe in this article is a relatively unknown but easy to scale Linux clustering solution, some interesting ways to run it, and use cases for the technology. Understanding past job scheduling technologies like these make it easier to understand why modern cloud computing is designed how it is.

Kerrighed

Kerrighed is one notable open-source example of an SSI implementation. In order to discuss Kerrighed, we need to discuss SMP - Symmetric Multiprocessing - first.

In the context of Linux, SMP - Symmetric Multiprocessing - is the technology through which Linux gains its ability to run on systems with multiple CPU cores or discrete processors, treating them as equals to share workload, memory, and I/O devices. In other words: when writing a program to run on Linux, you generally need not worry about how many or what kind of CPU cores the system running your program will have and how you will distribute your program’s workload across them. This is because Linux is in charge of scheduling processes or threads - that is, assigning them - to CPUs, and moving them to another CPU later if need be. Additionally, the word “symmetric” in SMP comes from the fact that no one CPU is the “leader”; rather they all work together in a cooperative way. This is SMP.

While Linux’s included SMP implementation runs on just a single system, Kerrighed ships as modifications to the Linux kernel that, when applied, implements SMP across many systems instead of one. Kerrighed is integrated in such a way that Linux tools previously meant to inspect just your local system - such as ps to list processes - now operate across the entire cluster. That is, ps would show you all processes running across all nodes in the Kerrighed cluster.

It goes deeper than that, too. A feature of SMP is that it will move processes to different cores depending on where resources are available. Kerrighed does this too, but the moves can be to a completely different physical system.

Because Kerrighed is implemented at the kernel level, this is completely transparent to the process being scheduled or moved. This is how Kerrighed achieves binary compatibility with existing Linux programs. This is Kerrighed’s killer feature - existing programs need no modification to take advantage of a Kerrighed cluster.

Diskless boot

Having a cluster of computers is great, but the challenge becomes managing the cluster. This section will discuss a different kind of “single image” - booting a fleet of diskless systems from a single operating system image, with the disk image being accessed as-needed over the network.

While diskless booting is not inherently related to SSI clustering, it complements the model nicely by making it easy to provision large numbers of identical nodes.

I first heard about this technique in the following tutorial articles:

A quick explanation of what these articles describe: all of the operating system’s root filesystem is shared via an NFS server. The network that these machines run on is configured with TFTP, a technology for booting over the network. When a host boots, it contacts the TFTP server and downloads initial configuration and a Linux kernel. These configs instruct the host to mount the NFS share, and to boot the operating system from the NFS share.

What is notable about the above approaches is how they handle writes to the filesystem. We can’t write to the NFS share under this setup because it is shared across however many servers are in the cluster. Instead, a special filesystem called AUFS comes into play. AUFS is a “layered” filesystem, in that it allows you to combine multiple “layers” into one coherent filesystem.

It is worth noting that AUFS inspired the more modern OverlayFS, which is probably a better choice today.

In this case, the bottom-most, immutable layer is our NFS share. On top of that, is an in-memory tmpfs filesystem. AUFS is instructed that writes should go to the tmpfs, and reads come from the NFS share. Of course, we want to be able to read back things that we wrote, and that is exactly how AUFS works: since the tmpfs is the “top” layer, a file present on it will be read instead of a file with an identical path on the “bottom” layer. This allows our network-booted operating system to write whatever it needs without modifying the NFS share and affecting other hosts.

Of course, memory is a sparse and valuable system resource, and consuming it for filesystem storage is a bit of a waste. AUFS does not need to use tmpfs; formatting an actual disk to a typical format like ext4 or xfs and using it as the “top” layer is an option too.

Combining it all

Now that we understand our components, it should be clear how we will create our SSI cluster:

Servers can be added to the cluster simply by plugging them into the network, enabling TFTP netboot, and turning them on. Each system will boot into our pre-configured system image and join the Kerrighed cluster. Kerrighed, by doing its task of emulating SMP, will balance processes onto newly joined servers and make their resources available to the workload running on the cluster.

The cluster is simple to manage: add machines simply by plugging them in and turning them on. Remove machines by shutting them down and doing the opposite. Update machines by configuring TFTP to point to a different NFS share with an updated system image and reboot each host.

Additional components may be needed for the cluster to be useful - such as mounting a read/write NFS share for workload data - but this covers the basics.

Use cases

Single System Image clustering is most useful for workloads that benefit from large amounts of parallel compute or memory but are not designed to run across multiple machines. Because SSI systems aggregate memory across nodes, the cluster can appear to applications as a system with a much larger pool of RAM. This can be useful for simulations, data processing tasks, or other applications that need large datasets in memory. This overlaps with High Performance Computing (HPC), which are generally scientific and engineering workloads. These are a natural fit for SSI systems as many of these programs are already written to take advantage of multiple CPU cores, but not necessarily multiple machines. An SSI cluster can allow these programs to scale beyond the limits of a single system by providing access to additional CPU cores across the cluster without requiring the software to be rewritten.

SSI clusters can also provide a degree of resilience. If a node becomes overloaded or unavailable, processes may be migrated to other machines in the cluster, allowing workloads to continue running. While this does not replace dedicated high-availability systems, it can help improve utilization and reduce the impact of individual node failures.

Modern comparable software

It was mentioned earlier on that Single System Image clustering is comparable to modern cloud computing. However, I refer only to Kerrighed in this comparison; the diskless boot design described above is orthogonal to SSI clustering.

Popcorn Linux seems to be the only open source, single-image clustering Linux distribution that is reasonably up to date. However, Popcorn is not binary-compatible with existing Linux programs like Kerrighed is. Instead, Popcorn uses a special LLVM-based compiler to compile programs in a specific way so that they may run on a Popcorn cluster. However, Popcorn also offers support for mixing different Instruction Set Architectures within a cluster and even supports migrating running processes to hosts with different instruction sets! Popcorn’s special compiler is what makes this possible.

Kubernetes is another modern comparable. Unlike Single System Image clustering, Kubernetes requires software to be packaged in a specific way - Open Container Initiative Images aka Docker Images - and “migrates” workloads by terminating them, moving required resources such as storage disks, and starting them anew on a new host. While Kubernetes has nothing to do with Single System Image clustering, it is similar in that its primary role is to schedule work on clustered hosts.

And that is what all of these tools do - Kerrighed, Popcorn Linux, Kubernetes - you give it your hardware and your desired workloads, and it figures out where to run it, and how to keep it online to the best of its ability. Single system image computing has its roots all the way back in the 1980s, whereas Kubernetes is relatively new, but both serve the same purpose, then and now.

If this interest you, other systems that explored similar ideas from over the years, include MOSIX and Sprite, both of which experimented with process migration and distributed operating system design.

finally TLS, courtesy of Let's Encrypt!

dave@davepedu.com (dave) — Thu, 20 Jul 2017 00:00:00 +0000

This blog - and some of my other sites - have finally been made secure, with some new TLS certificates courtesy of Let’s Encrypt. Browsing of my blog is now somewhat safer to network traffic inspection!

on moving to the Bay Area and engaging with Startups

dave@davepedu.com (dave) — Wed, 12 Jul 2017 00:00:00 +0000

I moved to the San Francisco Bay Area almost 3 years ago now. Having grown up in a small town (~2000 people) and gone to school in a small city (Rochester metropolitan area, New York ~1,000,000), it was not unexpectedly quite the shocking transition.

Most of my observations of difference are fairly mundane - traffic patterns are different, more timid in some ways, and more aggressive in others. Endless cool summer sun is easy to love. So many cultures and cuisines all in one place. A beautiful setting, and at the same time, it’s ground zero for the information technology revolution upending the world.

I saw what the Bay’s tech industry had to offer and found my niche over time. I came here to work for HP, an ancient giant in the hard and software industry, doing web development for marketing websites. I eventually found my way into the Startup Lottery, designing and operating critical core infrastructure for “big data” applications.

Startups are generally small companies that attempt to use computers and code to tackle challenges previously thought as too hard for such a small team. And the Lottery is the right term for how startups ebb and flow in the Bay.

Every so often, a “new thing” becomes the popular “new thing” and a thousand startups pop up around it. Doing “it” directly, supporting “it”, documenting “it”, consulting about “it”, defending from “it”, whatever “it” may be. The Dot-Com boom in the early 2000s, Web 2 and Social media in the later 2000s, early Apps in the 2010s, Big data in the mid 2010s, the list goes on. When it looks like there is not anyone left that wants to pay any more money for the old thing, a new thing pops up and the cycle begins anew. A handful of those startups may survive, and actually become profitable and valuable. And the few investors and few early employees in that company reap what is left of their particular thing, which is still substantially much, even though everyone is paying attention to the “new thing” now.

Once you understand that, you can see why working for a startup in Silicon Valley is like playing the lottery. You probably won’t, but there’s a slim chance you could pick the right company, stick with it, and be rewarded.

Even if that doesn’t happen, working for a startup is rewarding in other ways too. These companies understand they are inherently unstable, and as people generally prefer stable occupations over unstable ones, make up for it with anything from laid-back offices with free snacks to higher salaries. And as different types of perks appeal to different people, each startup tends to have a different personality.

As these companies are usually small, each employee is often in a greater position of responsibility. Large companies tend to let employees hyperfocus in specific areas and make slow changes involving a large number of people. Startups have fewer employees, which means each one must manage a larger slice of the company’s footprint.

In practice, this kind of environment pushes us engineers to think in terms of scale and leverage much earlier in their careers.

The only practical way to operate is to gain this leverage through automation. Tasks that might be handled manually in an older and larger organization quickly become unsustainable. You begin to look at systems not as one-off deployments or fixes, but as processes that must run reliably over and over again with minimal human involvement.

Automation is a force multiplier: the good decisions scale, but so do the bad ones. A poorly designed process that runs once might cause a small inconvenience. A poorly designed process that runs automatically across an entire fleet can cause a disaster in seconds. A process that works well in isolation may fail when invoked thousands of times simultaneously. Because of that, building automation forces a special kind of discipline. You have to research the tools you use, think through edge cases, make predictions with limited information and understand the systems you’re interacting with much more deeply than if you were just executing tasks by hand.

Over time, that kind of work develops a particular sort of experience. In larger organizations, thinking about systems at that scale is often the responsibility of architects or senior infrastructure teams. In a startup environment, those responsibilities fall much earlier in a person’s career. When a small group of engineers is responsible for keeping a large and complex system running, everyone ends up participating in the design decisions that shape how it works.

That puts you in an unusual position. You’re simultaneously operating the systems, building the automation around them, and deciding how they should evolve. The feedback loop between design and operation becomes very tight, or even nonexistent on a bad day. You feel the consequences of your decisions immediately, which encourages you to design things in ways that are easier to operate, monitor, repair, and repeat.

It also builds a habit of intentional experimentation. When your tools can affect large systems very quickly, you learn to test ideas in small steps. Changes get staged, rehearsed, and validated before they touch anywhere important. The process becomes less about heroics and more about methodical iteration.

In that sense, working in a startup environment isn’t just about “moving fast”. It’s about learning how to build systems that can move fast safely. And that combination of large responsibility, heavy reliance on automation, and constant experimentation creates a kind of practical architectural experience that would otherwise take many more years, and a much larger organization, to acquire.

And it’s fun to build and drive big toys using other people’s money.

In my current role, a large part of that philosophy shows up in the tools we create and use to operate our systems. I spend a lot of time working with tools like Ansible and SaltStack, both of which are designed to control and coordinate large groups of machines at once. Each brings its own model for turning a small amount of human effort into changes across an entire environment. Used well, they make it possible for our small team to operate infrastructure of surprising scale.

And honestly? All of this - the Lottery, the scale, the rewards - adds up to a whole lot of fun. The confident, but still nervous, satisfying feeling of seeing your change take effect on exponentially more systems is pretty addicting.

pfSense and ESXi setup on OVH's network

dave@davepedu.com (dave) — Mon, 12 Jun 2017 00:00:00 +0000

I host my projects on a dedicated server rented from OVH, I’m a huge VMware fan, so I’ve used their free product, the ESXi Hypervisor, to cram lots of virtual machines inside my one server. Since not all my virtual machines need internet-facing IPs and ESXi has only basic NAT capability, an easy solution here is running a pfSense install as a NAT router, as a virtual machine. OVH’s network has some unique setup required for this specific situation. Unfortunately, this is best documented in various blog posts across the internet.

Installation

You can grab the latest pfSense from here: https://www.pfsense.org/download/index.html. At the time of writing, the latest and pfSense version I’m using is 2.3.4, though any version later than 2.1 should work nicely. Thorough IPv6 support was introduced in 2.1, but if you don’t need that I suppose an earlier version would be fine as well. So grab an ISO and upload it to your ESXi machine.

Virtual machine creation

I used the 64-bit version of pfSense, so the “type” of virtual machine I used was Linux - Other Linux (64-bit). pfSense is based on BSD, so any similiar configuration should be fine. 512mb of ram allocated will be more than enough, but 256mb is doable. 4GB disk space allocated is fine as well. Next, other than adding a second network adapter, the default options that ESXi presents will work fine. One specialty for OVH - in your control panel, you must generate a MAC address for the address this host will be using - assign it to the public-facing network adapter. Also, create a virtual switch and attach it to the “private” network adapter. This switch is where your private network and virtual machines will live.

Installing pfSense to the virtual machine is pretty straightforward, but [check out this guide] (https://doc.pfsense.org/index.php/Installing_pfSense) if you need help.

OVH customization

After you finish pfSense’s command-line setup wizard, you won’t have internet connectivity. You need to add some routes specific to your server’s in the OVH network. If your server’s primary IP is A.B.C.D and the IP 1.2.3.4 will be assigned to this new host, run these commands in the shell:

route add -net A.B.C.254 1.2.3.4

route add default A.B.C.254

These commands will add routes needed to access your host’s gateway. You’ll have connectivity now, but need to make this fix permanent. Log into the UI, and in pfSense’s package repo the package Shellcmd is a convenient way to do this. Install it and add the two commands above as startup commands.

Wrap up

If you’ve enabled the DHCP server on your pfSense install you can now create privately-networked virtual machines with internet access via NAT, but with no exposure to the internet. Ports can be forwarded to public services and everything else is hidden behind a layer of NAT.

Next time: setting up IPv6 with TunnelBroker.net!

Editor’s Note: originally composed in 2014, but rotted in the drafts bin…

1Password on Linux

dave@davepedu.com (dave) — Tue, 28 Feb 2017 00:00:00 +0000

I’ve been a long time user of 1Password, a password vault by AgileBits. However, it only supports OS X and Windows; which is a problem for Linux desktop converts like myself. Their desktop application - a very nice app and also required for the browser extension - is only for Windows and OS X. AgileBits also offers 1Password Families which does support Linux, but you lose control over where your password data is stored.

In the past, 1Password had a browser-based user interface that could be served out of Dropbox or by opening the local file from your browser (called 1PasswordAnywhere, a single HTML file). However, browser security has “improved” and now block XHR requests to local filesystems and Dropbox discontinued the feature that allowed you to serve up this HTML file privately. So AgileBits was forced to sunset this feature.

Despite this announcement, 1PasswordAnywhere still seems to work! And as long as you can browse the file tree of 1Password’s data, this method should work for you.

I use Dropbox to sync my 1Password data, so I’ve opened a terminal and browsed to where the 1Password.agilekeychain directory is kept. For me, that’s ~/Dropbox/Apps/1Password/1Password.agilekeychain/. Within this directory is where 1PasswordAnywhere.html should be placed. I renamed mine to index.html.

Next, set up an web server to serve up this directory. If you’re running Mac or Linux you probably already have Python, and you can use its built-in minimal server:

$ pwd
/home/dave/Dropbox/Apps/1Password/1Password.agilekeychain
$ python -m SimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...

Next, browse to http://127.0.0.1:8000/ and you’ll be presented with a working - but visually outdated - 1PasswordAnywhere instance! Of course, this isn’t the most secure - anyone on your network can access the web interface too! To be fair - your password is needed to decrypt the data - though it would be better to serve only on localhost. But I’ll leave this as homework for the reader.

Also, first post of 2017!

Update: I also stumbled across gonepass, a native Linux desktop app intended to replace the 1Password app. I’ve not tried it yet, so expect a follow-up article!

Update 2: I downloaded and compiled gonepass on my system. It was pretty easy and works great! Definitely a safer idea than the web-based version above.

upgrading my NAS, part 2: high availability

dave@davepedu.com (dave) — Thu, 21 Jul 2016 00:00:00 +0000

In a previous post, I described some of my early approaches to building a NAS for my home. In summary, I decided to use Linux software raid formatted with Linux’s built-in ext3 format. This setup worked, but left me with three main problems:

Volume size limitations: ext3 is capable of handling a maximum volume size of 16TB, when formatted with a 4kb block size. It is theoretically possible to use a larger block size and thus a larger maximum volume size. However, changing the block size requires reformatting the volume, which I could not do. Additionally, most hard drives natively use a 4kb block size. Matching filesystem block size to hard drive block size generally achieves best performance, for reasons such as avoiding a single write operation becoming multiple operations - read-modify-write.
High Availability: I enjoyed having a NAS so much, that I quickly began running additional servers that would mount the NAS’s storage volume and use it for whatever large storage was needed. I can’t recall which network filesystem I was using at the time, but it was one that required a re-mount after the server rebooted. NFS would make this less of a problem. Anyway - reboots aside - I wanted a NAS that would remain available even after one of the computers backing it failed, or needed to be rebooted for updates, for example.
Space limitations: physical space becomes a limiting factor for NAS side as well. Not many server chassis had physical space for many hard drives, outside of rackable enterprise-grade systems that I couldn’t afford. Networks allow multiple computers to communicate, so why would my filesystem be limited to one machine? I wanted a NAS that could be “scaled up” by adding more computer servers and hard drives to it.

GlusterFS

Enter GlusterFS - GlusterFS is a distributed network filesystem. GlusterFS provides data safety features similar to RAID, but offered in a manner that spans multiple computers instead of multiple hard drives.

The general way in which GlusterFS functions is simple: you format your hard drives with whatever filesystem you prefer - I used ZFS - and GlusterFS creates a virtual filesystem on top of them. GlusterFS calls each sub-volume a “brick”, and while each brick only spans one server, the virtual GlusterFS filesystem can incorporate multiple bricks from multiple servers.

What’s great about GlusterFS’s approach to storage is that it single-handedly solves each of the 3 issues above:

Volume size limitations: GlusterFS uses its own type of filesystem with astronomically high volume size limitations.
High Availability: with a “replicated” GlusterFS volume, multiple copies of each file are stored on separate bricks. Since separate bricks are on separate servers, the file remains accessible even if one server or hard drive fails.
Space limitations: since GlusterFS volumes can span across multiple servers, space becomes nearly limitless. You can always plug more servers into the cluster.

Design

I started with two servers, but later moved to four. Each could hold up to 8 hard drives. A common build at the time was a mini-NAS, with a case by U-NAS, the NSC-800. For a motherboard/cpu combo, I chose AsRock Rack. I started with their Mini-ITX form factor server motherboard, the C2750D4I, and later upgraded to the C3758D4I. Both feature 8 cores, low power consumption, and small physical size.

As mentioned above, I used ZFS to manage my server’s hard drives. Initially, I had 16 drives spread across two servers. I divided these into 4 ZFS mirrors - like RAID 1 - of 2 drives each per server. Then, each ZFS mirror was replicated with GlusterFS to a corresponding ZFS mirror on the other server. This worked well, but I later moved to two ZFS RAIDZ pools per server, made from 4 drives each. This reduced the number of bricks which made GlusterFS simpler to manage.

Results

Overall, this was a fairly reliable and reasonably performant NAS. Generally speaking, I was bottlenecked by drive speed as I used “green” hard drives that put out less noise and heat and consumed less power at the expense of some performance. Sequential access was great but one area the NAS really suffered in was random access. Listing large directories was noticeably slow as well. Despite these issues, this NAS went on to serve me for many years!

If you run a similar setup, let me know!

building a distributed / redundant data storage backbone - part 1 of x

dave@davepedu.com (dave) — Mon, 23 Nov 2015 00:00:00 +0000

I’m not a fan of losing data, so I always have some sort of redundant storage at my disposal. At present I possess over 10TB of irreplaceable data, code, photographs, etc, (and much more replaceable data - though at an inconvenience) and housing that data is no easy task. Be it for backups, media storage, or even temp space, having a huge pool of “safe” storage is always a useful tool.

A bit of history first - we’ll get into the cutting edge stuff, with more technical detail, in part 2.

My first adventure into this involved a number of SATA hard drives attached to a Mac Mini via USB 3.0. A Linux VM assembled these drives into a RAID 5 array and served their contents over SMB (‘server message block’ - windows sharing). Unfortunately this was in the early days of USB 3.0 so the pass-through adapter to the Linux virtual machine only supported USB 2.0 speeds. So this array maxed out around 15MB/s constant write. Not great, but at least it’s safe.

RAID

Anyway, why RAID? RAID is a immensely valuable idea that allows the use of many independent hard or solid state drives for a single logical task. RAID can be used for speed gains beyond what an individual drive can handle or failure proof storage where even the death of one or more drives causes no data loss - and in the best case, no interruption to services using the data. There are many different kinds of traditional RAID, with the most commonly used being:

RAID 0 - performance gains by spreading load across multiple drives
RAID 1 - stability gains by mirroring data on two or more drives
RAID 5 - stability gains by smart use of striping & parity

These (and other) levels of RAID can also be combined for a mix of the effects a single RAID level brings. For example, RAID 10 (1+0, striped set of mirrors) or is fairly common.

Traditional RAID - or “hardware” RAID - typically involves a hardware expansion card for the dedicated purpose of handling RAID computations and drives. The RAID card would have a builtin processor, memory, and SATA ports to which the involved disks would be attached to. Up until recently, this sort of configuration was necessary for top performance - software RAID has been around for awhile, but historically there was a performance gap. A risk of hardware RAID has always been the RAID card itself. RAID cards typically have an onboard memory module a battery to keep the memory “alive”. If any of these parts cease to function - or any other part of the card - it’s possible that all data on the RAID array can be lost! In my opinion, this isn’t a viable option. If you have your data on a single drive or a RAID array controlled by one RAID card, either there is a single piece of hardware that can die and cause data loss. Not good.

Software RAID addresses this issue. It may have originally been created as a cheaper (and slower) alternative to hardware RAID, but I believe nowadays software options are just as valid and performant has hardware RAID is. Software RAID, in a nutshell, is, technically speaking, the same as hardware RAID, but with the computation and memory required for operation handled by a system’s main CPU and memory as opposed to a dedicated piece of hardware. Done right, these components can die or be swapped out with no fear of data loss.

In my opinion the most notable advances in software RAID tech are as follows:

Mdadm - often referred to as ‘Linux software RAID’ - was introduced in 2001 and quickly became the de facto software RAID tool.
ZFS - introduced in 2005 (and from my perspective, catching on in the early 2010s) was an entirely new beast.

While ZFS is impressively (and I’m using it today!), let’s examine mdadm first so we can see how ZFS differs. Mdadm offers block-level RAID. In other words, it doesn’t care which (and you have to use) filesystem is “on top” of it. On Linux, mdadm consumes block devices and provides a virtual block device. So if the physical drives on your system are /dev/sda and /dev/sdb, mdadm can consume these and provide /dev/md0. From there, md0 can be treated just like a plain block device - create partition(s) and format them with a filesystem of choice.

My first and second storage arrays used Mdadm. It was an easy choice as tools that work with native block devices work with mdadm virtual block devices. I could partition the virtual device with common disk management tools like fdisk and use common filesystems like ext.

And I did exactly that - my first array comprised of ext4 on top of mdadm. And it worked! …until I needed an array bigger than 16TB. Because of how ext4 was programmed and integrated into Linux operating systems it is incapable of handling filesystems greater in size than 16TB when using 4k blocks. Considering it originated in the early 90s and focused on backwards compatibility, there wasn’t a workaround. Back to the drawing board.

What’s nice about Linux is that filesystems are ubiquitous - if the system can mount a filesystem, all programs on the system that depend on file access and use it and all features it expects a filesystem to support. A beauty of modern system design. An up-and-coming filesystem around the time I needed to expand was btrfs. With a maximum supported filesystem size of 16EB (exabytes!), or a million times larger than ext4’s limits, I was sure I’d never outgrow it. So I live-converted my array to btrfs and it was full steam ahead from there!

Great! So in early 2011 I had a storage backbone capable of supporting an astronomical amount of data, the flexibility to expand over time, and safety of software RAID. What more could be desired? Well, the world of computing constantly improves itself so I wasn’t happy sticking with a single methodology. But more on that, and ZFS especially, in part 2!

subsonic database bloat & corruption

dave@davepedu.com (dave) — Sat, 01 Aug 2015 00:00:00 +0000

Update 04/08/2019:

I’ve had trouble with this method on newer versions of Subsonic and since the original technique was kind of bad - it more or less triggers crash recovery - I decided to take another look. I ended up finding the hsqldb-utils package, which includes some command-line utilities to manage HSQLDB databases.

Unfortunately, this package doesn’t use the same version of HSQLDB as Subsonic - my Subsonic instance is using 1.8.0.5 (it lists this in the database’s ‘subsonic.properties’ file), whereas the tools included in hsqldb-utils (on Ubuntu Bionic, at least) are well past 2.0. But, this is at least a step in the right direction. We can download an older version of the tools.

I found the following jars by scouring the internet, downloading them into /tools/ on my system.

hsqldb-sqltool-jdk1.5-2.0.jar
hsqldb-1.8.0.5.jar

Without further ado, here’s the happy-path way to de-bloat your database:

Install Java. You probably already have this since you’re running Subsonic, right?
Stop Subsonic. Subsonic must NOT be running while is operation is performed or database corruption is likely.
Change directories into the database dir. The path below could be different for you:
- cd /var/subsonic/db/
Open the database using hsqldb-sqltool. Take note of the classpath (-cp), this needs to be pointed at the two jars above:
- java -cp :/tools/hsqldb-sqltool-jdk1.5-2.0.jar:/tools/hsqldb-1.8.0.5.jar org.hsqldb.cmdline.SqlTool --inlineRc=url=jdbc:hsqldb:file:subsonic --driver org.hsqldb.jdbcDriver
Run the compact function:
- sql> SHUTDOWN COMPACT;
Exit hsqldb-sqltool by pressing control-d.

At this point, Subsonic’s database should be drastically smaller. In my case:

Before:

# du -sh /srv/subsonic
3.2G /srv/subsonic

After:

# du -sh /srv/subsonic
14M /srv/subsonic

As insert/update/deletes queries are performed data moves around inside of HSQLDB’s data files. Over time, and similar to a hard drive, data becomes fragmented and the database file size grows despite much of the space within it being “not in use”. What we’ve done is rewrite the database file in an optimized way such that there is no “unused” space left within the file; all the data is contiguous. It will bloat again over time, but that’s a concern for later. Do take a look at HSQLDB’s docs on the subject.

I maintain a Dockerfile for Subsonic and will eventually add this database compression method as an automated task within the image. This should solve Subsonic database bloat once and for all! Edit: This is complete! As start of the startup procedure, my Docker image will compress the database at every startup.

Original article below. I highly recommend NOT doing this; it is here for posterity:

I use Subsonic media streamer install to stream my music library on my mobile phone, media center, laptop etc. Recently, my web UI and other client apps stopped working and I noticed the subsonic server was spewing database errors.

Ugly things like:

87431 [btpool0-6] WARN org.mortbay.log - Nested in org.springframework.dao.TransientDataAccessResourceException: PreparedStatementCallback; SQL [  ]:
java.sql.SQLException: java.io.IOException:** S1000 Data file size limit is reached in statement** [update media_file set ]
at org.hsqldb.jdbc.Util.throwError(Unknown Source)
at org.hsqldb.jdbc.jdbcPreparedStatement.executeUpdate(Unknown Source)
at org.springframework.jdbc.core.JdbcTemplate$2.doInPreparedStatement(JdbcTemplate.java:798)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:591)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:792)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:850)

…With hundreds and hundreds of lines in each stack trace. Subsonic is written in Java and uses a database called HSQLDB which has some sort of size-limiting upper limit that’s being hit. Since it is similar to SQLite, we can check the size of the database files on disk. Inspecting these revealed it had grown to 2 gigabytes! Considering it holds only a list of media files, a few playlists and some settings, there’s no reason it should be that large.

Buried in HSQLDB’s docs I found a description of it’s startup procedure, here. It turns out by simply changing the “modified” setting to “yes” in the database’s property file, the engine triggers a database recovery routine (which actually restores an internal backup). On Ubuntu linux, the database property file is /var/subsonic/db/subsonic.properties. After starting subsonic and letting it grind for a few minutes, I just needed to subsonic re-scan my media files and it was back in business! Not one setting or play count was lost. After the cleanup, the database is all of 44 MB. Nice!

I thought this was a pretty easy fix, but tricky to find. If anyone finds this helpful, Let me know!

the pi-rack

dave@davepedu.com (dave) — Fri, 01 May 2015 00:00:00 +0000

I recently acquired a low-end 3d printer (0.3mm extrusion head) and after a few practice prints I decided to put my new skills to the test. I wanted to create a Raspberry Pi case, but I thought that would be a little boring. So the idea I settled on was creating something akin to a blade server - an enclosure that takes care of power and LAN distribution, that houses several Pis. Overkill? Absolutely!

Armed with $10 calipers, a few RasPis to measure, and Sketchup Make, I created the Raspberry Pi Rack!

I posted many more photos, and even the source 3d model files you want to print your own, over on Hackaday.

The bill of materials isn’t crazy either:

4x 1/4" by 6" threaded rods e.g. from Lowes
16x 1/4" lock nuts
24x 1/4" by 1/2" nylon washers
8x 1/4" wing nuts (for easy disassembly - optional, can add 8x lock nuts instead
1x male molex connector (I salvaged mine from computer fan parts)
1x prototype board
1x 6-pin SIP header
1x 6-pin SIP header, 90˚ angle
2x 6-pin SIP header, straight
12x M to F breadboard jumper
12x F to F breadboard jumpers
5 volt Ethernet switch
3x screw + nut sets (these are to secure the switch, I can’t recall what I used. Something like a M3x15mm screw).
12x RJ45 (ethernet) headers
Cat5/e cable (a couple feet should do the trick).
ATX power supply. Any desktop computer has one of these.

Not counting the Raspberry Pis, all the supplies come in easily under $100. Not bad.

Happy hacking!

This project was featured on the Hackaday.com Homepage!

ch340g usb to ttl/serial for Mac

dave@davepedu.com (dave) — Wed, 08 Apr 2015 00:00:00 +0000

I bought some USB to TTL adapters on the cheap from China. The OS X driver was a little difficult to track down, but this thread cracked it: http://forum.arduino.cc/index.php?topic=261375.0:

Okay, drivers for mac do exist. I always see www.wch-ic.com for the driver, but now I searched for “ch341ser_mac.zip” and found www.wch.cn The mac driver is here : http://www.wch.cn/downloads.php?name=pro&proid=178 The version is from wch.cn is a newer, you should try that one.

And the second piece:

Just to summarise. On Yosemite:

install the driver

Run this [in a terminal]: sudo nvram boot-args="kext-dev-mode=1"

Aaand it works!

Also known as /dev/tty.wchusbserial1410

Edit: Because I keep forgetting what settings to use in the Arduino IDE for this programmer, I’ve added them. Note that these settings are for this homebrew arduino, which supports the ATmega168 and ATmega328. I’m using the cheaper 168, but remember to use the right setting if you’ve got the faster chip.

Tools > Board: Arduino Pro or Pro Mini
Tools > Processor: ATmega168 (5v, 16mhz)
Programmer: AVR ISP

arduino pong, part 1: the display

dave@davepedu.com (dave) — Tue, 17 Mar 2015 00:00:00 +0000

As mentioned in my last post, I’ve been learning up electronics with the Arduino. I found a few 5 by 7 LED matrices at a local surplus store, pictured below, so I figured another fun beginner project would be creating a clone of pong using a few of these for the screen.

After some quick research - googling the part number - I found a datasheet, read it, and learned that the LEDs in these units are arranged in a grid pattern, with rows sharing anodes and columns sharing cathodes. Meaning, to light up all the LEDs in a single row, we provide power to only that row and ground all seven columns. The way this is designed, we can decide what LEDs we want lit in only a single row (or column) at a time. So, when an image is displayed using the entire matrix what actually happens is a single column/row is lit up at a single instant and this is applied to reach row as are scanned down through hundreds of times per second, so it looks to the human eye that the LEDs are on constantly. This is called the persistence of vision effect. Another nice advantage of this is power savings - even though an LED in the grid appears to be on constantly, it’s actually only on for 1/5th of real time (or 1/7ths depending on how you want to use the matrix - scanning by row or column).

I wanted a bigger screen than a single 5 by 7 matrix, so I’m stacking two right next to each other to form a 10 by 7 display. Since we now have 10 rows, we have 10 input pins. I can tie the ground (output) connection of correlating columns together so the entire screen still has only 7 ground pins. So, however we choose to run the screen’s logic, we will need to “push” 17 bits to the screen - 10 to describe the input state, and 7 to describe the output state. Let’s focus on setting that up first.

I decided to use a 74HC shift register, pictured on the right above. This chip provides 8 outputs that we can control using just three inputs. The shift register has an internal state, known as a register. When we “push” bits into the chip they are added at the beginning of the register, and existing bits cascade down (bit 0 moves to position 1, bit 1 moves to position 2, etc. The bit in the last position is dropped). Our data bits are signaled at pin 14, known as SI or “serial data input”.

To tell the chip that there’s a bit waiting on SI, we change pin 12 (RCK or “storage register clock input”) from 0v (low) to 5v (high). The chip recognizes this change and copies the single bit from SI to the internal register. Then, when we want the chips’ 8 output pins to reflect the state of the internal register, we simply change pin 11 (SCK or “shift register clock input”) to high. Again, the chip recognizes this change and updates the output.

There’s also pins to clear the register or toggle the chips ability to output anything at all but for we don’t need that functionality. They can be ignored by either tying them to ground or a 5v source, whichever the chip demands.

However, we will use another feature of the 74HC, located on pin 9 (QH or “serial data output”). When the 8ths bit is cascaded and “dropped” off the chip, it’s actually sent out this pin. If the bit is 1, this pin will go high for an instant, and stay LOW for a 0. This is exactly the kind of input the chip expects on pin 14, SI! This lets us chain as many shift registers as we want - QH of the leading chip to SI of the following chip, rinse and repeat. We also need to connect RCK and SCK of all the shift registers used, as the signal to read the next bit and display state all need to happen in sync.

Since I need 17 bits to control my screen, I’ll need to chain 3 shift registers. On a breadboard, that looks a like this:

Quick explanation:

The dense mixed color wires across the bottom are outputs, as well as the two green wires, and the grey one in column 57.
The vertical orange wires provide +5v to the chips power input and other pins that need a constant 5v.
The yellow wires ground pins called “Enable” which literally enable the chip’s functionality.
Pin 8 (lower right of each chip) is grounded (not pictured).
The three blue wires in the upper right are SI, RCK, and SCK (left to right). These would go to our microcontroller.
The horizontal orange wires is connecting QH of the leading chip to SI of the following chip (two instances of these).
The horizontal grey wires connect the RCK pins of all 3 chips together, and the SCKs all together.

We now have the ability to selectively provide power to the 10 rows of our matrix, but what about grounding the 7 columns? We need to be able to translate the 5v signal from 7 of our bits into a some logic: 5v means ground the corresponding column, and 0v means leaving the column unconnected. A basic way to do this would be a relay. A relay is like a pushbutton, but to “hold” the switch on we provide current to the relay, which it forces through a coil to create a magnetic field pulling two pieces of metal together to complete a circuit.

However, there are two problems that prevents using a relay for this case: Relays are slow: it physically moves a piece of metal to connect the circuit. We need to toggle the state hundreds of times per second; this would wear out a relay very quickly, if one could even keep up. Second problem, relays require higher current to trigger. More than we can safely provide from the 74HC registers at least. Sure, we could toggle a transistor using our register, and use that to control a strong power source, but that’s unnecessary complication.

The solution: Darlington pairs. These are simply two transistors connected in a way where we can toggle grounding of one input by providing voltage to the other. Transistors require much less power than relays, and can act much faster. We have 7 outputs to control so we’d need 14 transistors in 7 pairs. Unfortunately this would take up a large amount of physical space, at least 21 rows on our breadboard. Too much.

Enter the ULN2803 (left, pictured above). This chip contains 8 Darlington pairs is one tiny package - perfect! Mine happens to be a Toshiba brand, but they come in many, many flavors. We simply connect the 7 outputs from our screen to pins 12-18, and 7 connections from our shift register bank to input pins 1-7. And, most importantly, ground pin 9. Now, to write some code!

#define DS_PIN 2
#define STCP_PIN 10
#define SHCP_PIN 11
#define NUMREG 17
boolean registers[NUMREG];
boolean screen[10][7] = {
{1,1,1,1,1,1,1},
{1,0,0,0,0,0,1},
{1,0,0,0,0,0,1},
{1,0,0,0,0,0,1},
{1,1,1,1,1,1,1},
{1,0,0,0,0,0,0},
{1,0,0,0,0,0,0},
{1,0,0,0,0,0,0},
{1,0,0,0,0,0,0},
{1,0,0,0,0,0,0}
};
void setup() {
// Mark 3 pins for output
pinMode(DS_PIN, OUTPUT);
pinMode(STCP_PIN, OUTPUT);
pinMode(SHCP_PIN, OUTPUT);
}
void writereg() {
// Writes the 17 registers to the shift register bank
digitalWrite(STCP_PIN, LOW);
for(int i=NUMREG-1;i>=0;i--) {
digitalWrite(SHCP_PIN, LOW);
digitalWrite(DS_PIN, registers[i]);
digitalWrite(SHCP_PIN, HIGH);
}
digitalWrite(STCP_PIN, HIGH);
}
void writescreen() {
// Reads the screen array and manipulates regsters[] for each row
// For each column in the row
for(int i=0;i<=10;i++) {
// First, blank the registers
for(int rs=0;rs<17;rs++) registers[rs]=false;
// Mark the column on
registers[9-i]=true;
for(int r=0;r<7;r++) {
// mark any leds on
if(screen[i][r]) registers[r+10]=true;
}
// Push the result to the screen
writereg();
// Leave this frame on for a short time
delayMicroseconds(100);
}
}
void loop() {
// Write one frame to the screen, forever
writescreen();
}

This code is pretty simple too. First, we define which pins are what. Then, create an array of 17 booleans that will be what we manipulate in the chips memory before pushing it out to the shift registers. Our VRAM, if you will. In my setup, the first 10 bits represent row state, and the last 7 represent column state. For ease of use, we’ll create a 10 by 7 multidimensional array of booleans, 1 to 1 scale with LEDs on our screen. I set it up to show a “P” (for pong!) by default, and to confirm all our rows and columns are connected and working.

In setup(), we tell the arduino which pins are which. Writereg()’s purpose is to push our in-memory array of 17 bits out to the shift registers. Writescreen() calls writereg() once per row of the screen, for one screen “refresh”. Loop() calls this forever so we see the P on our screen.

Bingo!

Coming up in part two: Tracking user input!

nintendo 64 joystick pinout for arduino

dave@davepedu.com (dave) — Wed, 11 Mar 2015 00:00:00 +0000

I’ve been learning electronics and playing with an Arduino lately. I realized today I have a functional, but worn mostly out, Nintendo 64 joystick. For an electronics beginner like me, it would be a fun challenge to figure out how it works.

On a side note: I have a spare joystick because I replaced the ones in my Nintendo 64 controllers with a redesigned and improved replacement stick. The usual getting spots carry such a replacement joystick module for the Nintendo 64. It is engineered better than the original - it should never wear out!

Inside of the joystick module, is nice and simple single-sided board. I couldn’t conjure up a datasheet from googling markings on any single component on the inside, so we’re on our own! The circuitry inside looked like this:

Simply put, pins 2 and 3 are power and ground (respectively), 1 & 4 are data for left/right, and 5 & 6 for up/down. I measured approximately 3.3 volts across pins 2 & 3 in a working controller, so we’re in luck! The arduino provides a 3.3v rail. I measured that the joystick circuit pulls about 15mA, so we’re more than in the clear to power it from the Arduino. R1 / R2 are 320Ω; I didn’t have equipment to measure C1 / C2.

In the image above, two slotted discs pass through the cutouts near U1 and U2, one for each axis. The movement of the joystick is geared to each slotted disk, so the precision actually looks pretty high.

I also don’t own an oscilloscope so figuring out the format that the sensors return data could be pretty tricky - guesswork, really. But again, we’re in luck! Someone’s created an easy way to use the arduino as a basic oscilloscope! Aren’t general purpose processors awesome?

However, viewing the output from any data pin all looked the same - a flat HIGH or LOW signal when the stick is moving, and rapid oscilation between high and low during motion. The demo code only supports a single input. Theoretically, it could be expanded to 6 as the Arduino contains 6 analog inputs, but I’m not wanting to dive into that yet. After some research, I found that the sensors in the joystick appear to be two linear sensors! This kind of sensor uses optics to track when a slotted medium interrupts some beams of light and encodes the result into two waveforms who’s period indicates speed of motion and offset from eachother determines direction of motion. This video does a great job of explaining it, and even provides sample code.

So, how to use this component with an Arduino? Pins 2 & 3 can be connected directly to the 3.3v rail & ground. 1 & 4 can go to any digital inputs; 5 & 6 must go to digital inputs that support interrupts.

I modified code from the video linked above to create the following:

#define encoderIx 2
#define encoderQx 3
#define encoderIy 18
#define encoderQy 19
volatile int countx;
volatile int county;
void setup() {
Serial.begin(9600);
countx=0;
county=0;
pinMode(encoderIx, INPUT);
pinMode(encoderQx, INPUT);
pinMode(encoderIy, INPUT);
pinMode(encoderQy, INPUT);
attachInterrupt(0, handleEncoderX, CHANGE);
attachInterrupt(5, handleEncoderY, CHANGE);
}
void loop() {
Serial.print("X: ");
Serial.print(countx);
Serial.print(" Y: ");
Serial.println(county);
delay(10);
}
void handleEncoderX() {
if(digitalRead(encoderIx) == digitalRead(encoderQx)) {
countx++;
} else {
countx--;
}
}
void handleEncoderY() {
if(digitalRead(encoderIy) == digitalRead(encoderQy)) {
county++;
} else {
county--;
}
}

How this works is pretty simple - X and Y are handled separately, and stored in two ints. We’ll look at just a single axis. When the signal to the interrupt-supporting pin is changed, it is compared to the other pin on the same axis. The joystick’s linear encoders express forward/backward as one wave lagging or leading the other - and at the instant the interrupt function is ran, comparing the interrupt pin to the 2nd input from the same sensor allows us to determine this. Depending if the other input is high or low, we either increment or decrement the position counter for this axis.

The sample code above will print the X and Y position of the stick to the arduino’s serial console. I used an arduino mega with the code above; the interrupt-supporting pin numbers might not necessarily be the same for other arduino models.

Build anything with this? Let me know, and I’ll post it here.

Happy hacking!

coast to coast

dave@davepedu.com (dave) — Thu, 30 Oct 2014 00:00:00 +0000

In 2014, I graduated from the Rochester Institute of Technology, and shortly moved to the Bay Area, California. I drove my car, filled with all of my possessions, from New York to California. These are some of the sights I saw along the way.

First stop!

Our first stop was at the Indianapolis Motor Speedway Museum, where we saw the 1911 Marmon “Wasp”, winner of the first Indianapolis 500, as well as a few other classic race cars.

Kansas

…Is less flat than I was led to believe, but still pretty flat. And thousands of these windmills.

Castle Rock, CO

Our third stop. An adorable town south of Denver.

Climbing the rock

We climbed the ‘Castle Rock’. Great view of what was going on in town.

Looking west

From the top of castle rock, we could clearly see the Rockies looming in the distance, snow peaked in June. We’re going over that.

The peaks

We stopped in one of the mountain towns, along our approach to the summit, for gas, near the tipping point.

Nature in the Rockies

Can’t beat highway rest stops with a view like this.

Cliff

I think this was on the west side of the Rockies.

Utah

Made it into Utah. Amazing how much more dry it was. (And my awesome car!)

More Utah

Crazy rocks in Utah

Salt flats in Nevada

Flats isn’t an exaggeration. Hard to walk on, too. It’s difficult like beach sand, but slippery and sticky.

Honda

Another shot :-)

2 days later, in California

Emigrant Gap

Southwest of the bay

Looking directly at the area I’ll be living/working in!

Pacific Ocean

Finally ran out of road.

Beach Cave

Obligatory Golden Gate photo

You knew it was coming.

Learn2bridge

Downtown San Francisco

Shot from Coit tower

Lombard Street

You can bet I went and drove down it after.

Santa Cruz

South of the Bay

My new office!

Yeah buddy.

All in all, a wonderful drive that took about a week. I was lucky enough to have my father come along with my for the trip. It was a lot of fun, and it was great to see this beautiful country.

internet radio DVR

dave@davepedu.com (dave) — Sat, 30 Aug 2014 00:00:00 +0000

Anyone who remembers watching cable TV also remembers the DVR - Digital Video Recorder - a device that lets you record live TV and save it for later playback. It’s useful for catching episodes of shows that air when you are busy or sporting events overseas that air late at night for your timezone.

Well, I listen to a handful of shows that only appear on live internet radio, and I wanted a similar way to record and play them back later. So, I wrote this application.

This application provides a web interface where you may configure audio stream URLs and the schedules at which they should be recorded. Each recording is saved as an MP3 file, and an RSS feed that can be consumed by any Podcast player app provides easy access.

Under the hood, this application uses FFmpeg to do most of the media recording and manipulation.

Source code is available on my Git instance:

Base application: https://git.davepedu.com/dave/streamrecord

Docker image: https://git.davepedu.com/dave/docker-streamrecord

coda 2, git, and drupal

dave@davepedu.com (dave) — Wed, 21 May 2014 00:00:00 +0000

Coda 2 (Previously, just Coda) has been my “daily driver” suite for web development for around 4 years now. It’s an absolutely wonderful editor by Panic, and well worth the price tag in my opinion. It has an editor, (s)ftp client, git client, terminal, mysql database manager, and a whole lot more all in one package.

My workflow when developing a Drupal-based website is usually something like this:

Create a new drupal install on my local machine.
Build the theme and database setup locally
When it’s ready for a demo or for whatever displaying, transfer the theme & site files to (typically) the multisite install where the site will live. This usually means manually transferring the local database as well. If you’re familiar with drupal you’ll know this means tracking down and changing Drupal’s internal filesystem settings.
Repeat steps 2 & 3 until the site is done.

I wanted to remove some of the manual steps, and use Git for version control. I use my own git instance, but you can also use Bitbucket. So, here’s how to do that in Coda 2! This is based on the assumption you already have a site started, if you’re starting from scratch it would be the same procedure, though. I’ll also assume you’re familiar with at least the basics of using a unix/linux command line.

Set up a Local Git Repository for the Drupal site

I’ll admit I’m not great at using command line git, and have usually stuck with using graphical clients such as Gitbox. But I promise, after this initial setup you can stick with the same tools!

First, in a terminal, change directory to your site’s main folder. This would be it’s directory in the drupal Sites folder, that’s typically named after your domain. I’ll be using this blog’s setup as an example. We will first create a .gitignore file so items that don’t need to be tracked (such as configuration and temporary files) won’t be. My .gitignore typically looks like:

files/css
files/ctools
files/custom_search
files/js
files/less
files/styles
settings.php

Including settings.php is the crucial thing here, this is where the database password is stored! Now that it is being kept secret, we can create the local git repository:

git init

This created a basic git repository based in the current directory.

git remote add origin https://dpedu@bitbucket.org/dpedu/projectname.git

Tell git that this project’s origin (or master repository, so to speak) is located on Bitbucket. Substituting Github or another git host is doable.

git add --all

Add all existing files to version control

git commit -m 'Initial commit'

Commit all existing files

git push -u origin --all

Transfer the commit we just made to the origin, aka Bitbucket.

Great! Now we have a local Git repository and a master one on a hosted Git service.

Set up Coda 2 to use our repositories

The Coda setup is pretty typical. You want to set Local Root to the same directory where we just made a local repository, though.

Next, click the Source icon.

On this screen, you’ll only need to change Off to On, select origin in the dropdown, and put your Bitbucket password in the password field. Coda should automatically know the git URL, as it’s saved in the repository’s metdata.

Great! You can save this profile and get to work!

Using Coda

Once you open up the profile we just created, you’ll see the local filesystem and remote filesystem within Coda. I like to use the righthand sidebar for just about everything. The files tab shows your theme files, the version control tab (Coda calls it “SCM”) is used for working with git, and the publishing tab is used for pushing your local files to the remote server. You can set up git hooks to do this pushing automatically, but I prefer not to when working with Drupal.

After you save changes to a file in Coda, it will be listed in the version control and publishing tab, along with a counter the number of files changed can be seen at a glance.

Right clicking on the file reveals options to:

Compare it to the latest or any revison using FileMerge, a tool shipped with OS X.
Add it to the next commit
Commit it right now
Revert, branch, merge and all the other git actions.

Adding & Committing it work exactly how you would think. At the bottom of this sidebar is the button to push any local commits to the origin (internet-host hosted) repository.

Once you select this, your committed changes are pushed to bitbucket!

Cool! Your development setup is now fully integrated with git. Want to push your local version to the remote web server, configured when settng up the site profile in Coda? The publishing tab has the option to Publish All, which uploads any changed files.

The database

Drupal heavily relies on a database for all of it’s content, and probably far too much configuration. Moving your local database to a live server can be a bit of a pain, so here’s a little script I use to help make it easier:

#!/bin/sh
# Dump database
/Applications/XAMPP/xamppfiles/bin/mysqldump --hex-blob --extended-insert=FALSE -u localmysqluser -plocalmysqlpassword localdatabasename > dump_raw.sql
# Remove cache insert queries
sed '/^INSERT INTO `drupal_cache/ d' < dump_raw.sql > dump.1.sql
# Remove watchdog insert queries
sed '/^INSERT INTO `drupal_watchdog`/ d' < dump.1.sql > dump.sql
# Remove temp files created above
rm dump_raw.sql dump.1.sql
# Append queries to update some drupal settings to values that will work on the live server
echo "UPDATE \`drupal_variable\` SET \`value\`='s:27:\\\"sites/dpedu.io/privatefiles\\\";' WHERE \`name\`='file_private_path';" >> dump.sql
echo "UPDATE \`drupal_variable\` SET \`value\`='s:20:\\\"sites/dpedu.io/files\\\";' WHERE \`name\`='file_public_path';" >> dump.sql
echo "UPDATE \`drupal_variable\` SET \`value\`='s:4:\\\"/tmp\\\";' WHERE \`name\`='file_temporary_path';" >> dump.sql
# Transfer SQL dump file to my homedir on saturn
scp dump.sql dave@dpedu.io:~/
# Connect to server, import the dump file, delete the file on server
ssh dave@dpedu.io 'mysql -u remotemysqluser remotedatabasename < dump.sql ; rm dump.sql'
# Delete local file
rm dump.sql
# Done !
echo "Done"

Neat! So what does this do?

Dumps your local database to ‘dump_raw.sql’.
Using sed (and some temporary files), removes inserts to any of the various drupal_cache and drupal_watchdog tables. The cache tables need to be cleared in such a move anyways, and the watchdog log messages really aren’t too vital and often more than double the dump size.
Adds 3 queries to update drupal’s internal filesystem information to be executed after the database is imported.
Transfers the sql dump file to the remote server. My SQL server isn’t accessible from the internet. Security!
On the remote server, initiates an import of the dump file.
Cleans up after itself.
So now your local install is perfectly mirrored, within seconds.

Found this useful? Have a better setup? Write to me to brag about it.

RIT Project Lead The Way Website

dave@davepedu.com (dave) — Sun, 30 Mar 2014 00:00:00 +0000

A website Project Lead The Way Website, where the Rochester Institute of Technology serves as New York State’s National Affiliate.

This website has since been decommissioned and replaced with an updated design, but my work may still be viewed via archive.org.

RIT University Arenas Website

dave@davepedu.com (dave) — Fri, 28 Mar 2014 00:00:00 +0000

A massive Drupal website, that took us nearly two months to design and develop! It’s fully responsive, has some great animation and attention to detail, and a good number of photos from previous events at RIT.

This website has since been decommissioned and replaced with an updated design, but my work may still be viewed via archive.org.

RIT Board of Trustees Intranet Portal

dave@davepedu.com (dave) — Tue, 28 Jan 2014 00:00:00 +0000

The RIT Board of Trustees Intranet Portal is a CodeIgniter based web application that I designed and developed. It is an intranet website that is used to disseminate meeting materials for the Rochester Institute of Technology’s Board of Trustees meetings, as well as poll its members for attendance, feedback, etc.

Code from this project was re-used for another project, the RIT President’s Roundtable, an advisory committee for the President of RIT.

RIT Center for Urban Entrepreneurship Website

dave@davepedu.com (dave) — Tue, 28 Jan 2014 00:00:00 +0000

Another beautiful Drupal website, this time for RIT’s Center for Urban Entrepreneurship in downtown Rochester.

This one was designed in-house at RIT Production Services, with a nice Responsive theme for mobile phones and tablets as well.

This website has since been decommissioned and replaced with an updated design, but my work may still be viewed via archive.org.

kennedy space center, cape canaveral, florida

dave@davepedu.com (dave) — Fri, 17 Jan 2014 00:00:00 +0000

Last November, I had the opportunity to visit the Kennedy Space Center in Florida, while visiting relatives living in the area. I had never been there before, so I went for it!

I may be a tech geek, but these amazing machines fascinate me. The sheer amount of research, time, and trial-and-error fueling these machines is simply incredible. Staring at the results of the most advanced transportation machines made by mankind is mind boggling.

If you have any interested in space exploration or that sort of thing, check it out!

Entry

Right at the entrance, we were greeted by an array of rockets used over the years. The specific names of all the models escape me, but the largest one is the legendary Saturn 5 model - the most powerful machine ever built by man.

Motor Closeup

The business end of this rocket looks more like a tangled mess of tubes and wires, but its a very advanced and complicated machine.

Stone Ball Skymap

Instead of a boring fountain, the center had a map of the cosmos carved into a 2 ton granite sphere, suspended and slowly rotating on a water base. With enough force you could spin it a different direction!

Atlantis Base

The solid state rockets and liquid fuel tank for the Atlantis Space shuttle, The picture doesn’t do any justice to just how massive this is.

Facility Panorama

A panorma with a view of several facilities. The location this was shot from was formerly a public viewing area for space launches, but has been deemed too dangerous since then.

Click the image to see full-size for a better view.

Crawler

One of NASA’s two land crawlers, currently being outfitted for SpaceX missions.

Vehicle Assembly Plant

This building is where every NASA space vehicle is assembled. Again, the picture does no justice to the height of this building.

Another Crawler

Another one of NASA’s land crawlers, hiding inside.

Seaside Bunker

Looking out into the Atlantic, one of the main launch platforms would have been behind me. According to our guide this little bunker was used to house sensitive equipment for detecting electricity in the air, as lightning is a grave threat to launches.

Lightning Towers

3 lightning towers designed to direct bolts away from the launch pad that they surround.

Saturn 5 Rocket Base

The business end of a decomissioned Saturn 5. Walking the length of this rocket takes minutes! 20 tons of fuel per minute would flow through here.

Saturn 5, second stage

Command Module

Apollo Guidelines

A list of mission status and abort codes from the Apollo era. Click the image to see full-size for an easier read.

Space Shuttle Atlantis

The retired Space Shuttle Atlantis. 🇺🇸 ‘Murica!

Birds!

That’s the last of the space pictures, I took a few quick shots of birds outside my hotel a few days later on the same trip.

A Pelican

“What you lookin’ at?”

Lucky Shot!

A gull dive-bombing to catch a fish, at the moment of impact. Took quite a few trys to capture this!

RIT KGCOE Website

dave@davepedu.com (dave) — Sun, 01 Dec 2013 00:00:00 +0000

This was actually a project I completed about a year prior, but we decided to revisit it to update the site and add responsive design features - try the site on a smartphone, it’s excellent!

The layout was designed in house and I did 100% of the development work. The old layout dated back to 2002, so it was obvious this site needed a refresh! It’s amazing how much different the web looks, now.

The KGCOE website was the first of a new family of websites - or a group of websites with a similar look and feel. The layout was adapted for several more Engineering websites at RIT, including Computer Engineering and Mechanical Engineering. These sites also share some content - Staff and facility listings, for example - and that was interesting to put together.

Overall, I’m pretty happy with how the site came out, and working with the school’s Engineering department was great!

Additionally, this website won a Silver Award, given by The w3 Awards in its Best of the Web competition.

The new College of Engineering website was honored for its homepage design, the structure and navigation of the site as well as its visual appeal. Undergoing a complete redesign in the past academic year, the site incorporates multiple carousel images and stories of student and faculty work, separate informational sections for prospective students and corporate partners, testimonials as well as in-depth information about majors and programs offered in the college. Contributors for the site include: Bhalla; Bastian; David Pedu, Web programmer; and Karen Ester, communications specialist in the Kate Gleason College of Engineering.

Two of our Web projects were recognized by W3 last year, so it is great to get continued recognition from them this year as well. It’s a testament to the immense talent that we have internally at RIT,” said Raman Bhalla, assistant director, Web/IT services.

- RIT News

This website has since been decommissioned and replaced with an updated design, but my work may still be viewed via archive.org.

RIT Innovative Learning Institute Website

dave@davepedu.com (dave) — Sun, 24 Nov 2013 00:00:00 +0000

The Innovative Learning Institute at RIT is a new department, under the Center for Multidisciplinary Studies branch. The layout is very similar to the CMS website, and was also designed by Dumbwaiter Design.

This website has since been decommissioned and replaced with an updated design, but my work may still be viewed via archive.org.

oktoberfest 2013

dave@davepedu.com (dave) — Thu, 17 Oct 2013 00:00:00 +0000

Every October, the ski resort at Whiteface Mountain has a small Oktoberfest. Traditional German music, food, beer, and demonstrations are featured, and many local vendors come to show off their homemade crafts.

This is the first time I’ve gone in many years, and the first time I went being older than 21! It’s a ton of fun and is definitely worth checking out if you’re in the area during the skiing offseason.

The atmosphere at these events is always awesome, it’s friendly and inviting. And I’ll never forget the angry-sounding accordionist saying “Drink the beer, Now” in his thick accent.

Whiteface Lodge

Deck of the Whiteface Lodge, at the foot of Little Whiteface and Whiteface mountains

Grounds

Looking out over the grounds of the area, many local vendors set up tents to show off their homemade products every year.

Cloudsplitter Gondola

The quickest way to the top of Little Whiteface!

Yodelers

A group of 3 yodelers and their alphorns, these instruments were traditionally used for communicating mountaintop to mountaintop in the past.

Yodeler Show

Inside the Whiteface Lodge, the Yodelers teamed up with a traditional German band, DSB die Schlauberger.

Traditional Accordion Player

I really wish this photo came out better. Oh well!

Selfie!

This guy was awesome.

Up the cloudsplitter!

Can’t beat that view! This is why I love the Adirondacks.

Looking down

Halfway up the mountain, looking down.

Base camp

A better view of the festivities below. The line across the picture is the arm-size cable hauling our gondola to the summit.

Almost to the top!

Nearly there! And an amazing view of the surrounding mountains.

Yodeling

We met our yodeling friends from earlier, they were demonstrating how sound from their alphorns could be bounced off other mountains and heard echoing. Very neat demo!

Whiteface

Facing the opposite direction as the previous photo, the view of Whiteface from Little Whiteface.

Alphorns

The business end of some of these horns. Looking east.

Locking horns

Horn artwork

Very pretty!

Trails

Facing north, a good view of the ski trails connecting the two mountains.

Summit

A short distance from the gondola drop-off, the peak of Little Whiteface.

Tribute

A tribute to soldiers of a local division in WWII, a gift from Italy.

Olympic Ski Jumps

Not the best photo, but I think it came out alright considering the range was about 12 miles.

Olympic Bobsled Runs

Another long distance shot at about 8 miles, the Olympic Bobsled run at Mt. Van Hoevenberg.

Lake Placid

Facing west, the view of Lake Placid from the mountain.

Ticket

My “Cannon” and my ticket.

RIT Center for Multidisciplinary Studies Website

dave@davepedu.com (dave) — Mon, 14 Oct 2013 00:00:00 +0000

The CMS Website was created for RIT’s Center for Multidisciplenary Studies, and was graphically designed by Dumbwaiter Design and developed by myself. It’s based on Drupal to allow the CMS staff to easily create and post content to the website.

This project was a new challenge - typically, all of our designs were created in-house. Which is very convenient, if I need anything changed or clarification it’s a 10 second walk to talk to the designer. But, working with the CMS staff and Dumbwaiter Design went absolutely great!

This website has since been decommissioned and replaced with an updated design, but my work may still be viewed via archive.org.

exploring benson mines

dave@davepedu.com (dave) — Tue, 01 Oct 2013 00:00:00 +0000

Benson Mines is a long-abandoned iron mine near Watertown, NY.

The mine was shut down in 1978 and the buildings and have been left to nature ever since. It’s a great visualization of nature slowly reclaiming it’s land.

I stopped by with a friend on a rainy day on my way back to Rochester from the Adirondacks. The rain was moderate at times and the buildings were alive with the sound of dripping water.

Without further ado, the album:

Entrance

Entrance to the facility - and my car!

Machine Shop

Judging from the abundance of 440 volt hookups in the building, this likely was some sort of machine shop.

Safety for All

No Entrance

Back door to the machine shop.

Ore Silos

Rusting silo between concrete silos for holding iron ore. It looked like there was more machinery on top of the silos that had rusted away and collapsed.

Overgrown

Tree blocking a doorway.

Garage Door

Outside view of another building showing the decay of the area.

Burned Steps

Even older looking stairs - the mine has history back to the 1800s, but this is probably not that old. The building on top of it appears to have burned, leaving behind electric wires and boxes.

Abandoned Excavator

…With no bucket. Likely a Massey Ferguson 450.

Abandoned Excavator’s Engine

Machine Shop Roof

Outdoor view of the first building, the roof on this side is nearly rotted through. Most of the buildings were built with cinderblock walls, metal trusses, and wooden roofing.

Workshop

Inside another building, some sort of workshop.

Worker’s Desk

Transformer Gauge

The guts of the transformer were missing, probably stolen by copper thieves.

Office Building

One of the few two-story buildings, with major fire damage inside.

Another office

Some sort of office area, completely rotted out.

Bathroom and Sink

Likely decimated by vandals.

Outside

The office/bathroom is inside this building, and the first building in the album is visible through the trees in the background.

Exit (Entrance)

2025 Update

Benson Mines has had a complete transformation over the last couple years. The site was sold to a new owner, who demolished almost every building, clearing the site. More recently, the site has been transferred to a new developer who plans to create a 12-megawatt solar energy generation facility! While it appears Benson Mines’ days as a mine are over, it will live on as a new and futuristic facility.

test blog post

dave@davepedu.com (dave) — Mon, 30 Sep 2013 00:00:00 +0000

Hello! Welcome to my blog.

This is a test blog post to help develop the system.

RIT Wildenhain Exhibition website

dave@davepedu.com (dave) — Sun, 04 Nov 2012 00:00:00 +0000

A website showcasing the work of mid century artist Frans Wildenhain, built on WordPress. Features a prominent art photo gallery, and a WordPress blog.

Additionally, this website won a Silver Award, given by The w3 Awards in its Best of the Web competition.

The Frans Wildenhain Exhibition website was designed to feature the fall 2012 exhibition of mid-century modern pottery by Bauhaus-trained ceramist Frans Wildenhain.

“The site had to be designed to attract and hold the diverse interests of an international audience,” says Bruce Austin, professor of communication. “The efforts of the Web/IT team led to a website every bit as innovative, exciting and dynamic as the art on exhibit."

“These awards are a testament to the talented staff we have here at RIT,”" says Raman Bhalla, assistant director for Web/IT services. In addition to Bhalla, the Web/IT team includes Megan Bastian, Harry Groover, Gage LaFleur, Elizabeth Lamark, David Pedu and Dan Roder.

- RIT News

This website has since been decommissioned and replaced with an updated design, but my work may still be viewed via archive.org.

about this website

dave@davepedu.com (dave) — Mon, 01 Jan 0001 00:00:00 +0000

content

All text, images, and other media on this website are original works, unless otherwise noted. Any third-party material is used with attribution.

No content on this website was produced using Artificial Intelligence.

attributions

Site generated with Hugo.

Hugo theme adapted from re-Terminal by Mirus.

contact

dave@davepedu.com (dave) — Mon, 01 Jan 0001 00:00:00 +0000

direct

Email: web@davepedu.com

common

GitHub: @dpedu

LinkedIn: /davepedu

𝕏 / Twitter: @dpedu

Bluesky: @davepedu.com

Facebook: /dave.pedu

other

Strava: Link

Steam: moptop650

Last.fm: Moptop650

retired

Google+: Archive

Skype: d.pedu

feeds

RSS: Link