Z Space Blog - Organization

Refactoring My File Hierarchy

2020-06-20T21:06:00-07:00

When I returned to mkbak¹ in order to give it better options for custom backup naming, automatic date tagging, preservation of permissions, etc., I noticed that I was writing complex and tricky rsync filter files. I ran with this for a while -- and still have them as I write this -- but the problem stewed in my mind. Something bothered me about the organization of the files under my $HOME. Eventually, I felt there was a way for me to shorten my rsync filter files and reduce the stress I felt about my files. I just needed to figure out what the core problem was. To answer that, we need to understand what hierarchy's purpose is, or what its use introduces to your data.

Establishing the Branch Points

Academics can give you a more elegant explanation, but I understand hierarchy to be the way we divide and group objects of a type. Each split in the tree should be a meaningful distinction between groups of data. This distinction then facilitates easy browsing or finding of files. Of course, this is in theory, and to a certain point of nesting. If you're drilling down 17 levels, maybe try tagging! :P

So, to start with my backup file lists were getting complex. I added dot-folders, excluded cache folders and other transient files, made exceptions for in-git config files or in-mkbak, and mixed the origins of the files I was managing. That's when it hit me: I care about "I made this" or "someone else made this". That was the first meaningful division in my refactor. I treat files made by me as MUCH more important than those by others, though I still have local git copies of a lot of software I use, and distfiles for everything else.

Suddenly I noticed there was a lot to sort through. I had books and RFCs and specs mixed in with my articles, notes, and Ledger journals. I had icons mixed in with Pixel Art instead of being at the top level of Artwork. Some directories, like my music, weren't clear "Am I allowed to have this file?"... so there was another good distinction.

Next, I remembered that some music albums, games, or even software was bought by me, and due to the Copyright Act of 1976, one may keep a single archival backup of media, but only for personal use to protect your purchase. That meant I needed a place to put my dumped cartridges, Humble Bundle games, SoundCloud albums, retail albums, etc.

A Brief Breakdown

By this point I had four "buckets":

Mine
Others
Disallowed
Purchased

I couldn't think of another distinction that was big enough to belong at the top level, so I went with the four to start with.

The Mine tree covers files created or maintained directly by me. Stuff like family albums, projects, artwork, prose, etc.

The Others tree is for files made by others that are legal to have a personal copy of and/or are free to distribute in unmodified form. This includes code repositories, software binaries of freeware, etc.

Naturally, Disallowed just shows me what I need to delete, or purchase a copy if I liked it. Good to have separate. It was surprisingly much smaller than I expected.

Lastly, Purchased is an attempt to catalog and account for my purchased digital files, to prove their legitimacy.

Improving the Hierarchy's Usability

These initial groups already gave me insight into my file collection, but I thought about further classification or "features" I wanted in my hierarchy. I'm still shopping for a new OS; Where would I put VMs and their scripts? What about WINE-compatible installer binaries? I also wanted a "scratch area" that would store files that need to persist between boots², but aren't yet ready to be added to the hierarchy.

This part is actually much simpler for me, because I already organized my files by their type. Well, generally anyway. This was an opportunity to improve it. It wasn't that bad to basically swap directories around, rename a few things, update a few configuration files... With each directory I had sorted, I felt like the files were reaching a meaningful location for me that I didn't need to think too hard about.

Enter GNU Stow

That's when it hit me: "Didn't I hear about some way to map or merge directories, or manage symlinks?" I did, actually. GNU Stow is exactly that: you manage a set of 'packages' that roughly match up to directory trees and then merge them into wherever you want, to make it appear as if the files from all the others are in one place. Exactly what I needed. As a bonus, you can setup a given directory as STOW_DIR (I used /usr/local/stow) and then symlink your packages from there, so you aren't needlessly copying files or moving dirs around.

As a test of GNU Stow, I took music that was freely available and music I had purchased, which were in two different directories, symlinked to them in the STOW_DIR, merged their contents with GNU Stow, then hooked that directory up to real software to make sure it worked. Here's what it looked like:

/usr/local/stow/
  ├── music_others -> /home/zlg/others/music/
  └── music_purchased -> /home/zlg/me/purchases/music/

Then, I just ran a single command:

stow -d /usr/local/stow -t ~/music -S music_others music_purchased

After that, I did an ls ~/music and... son of a bitch, it worked! I double-checked by opening ncmpcpp and updating the mpd database, which was pointed to look at ~/music. The database looked the same as it did before. Playing files worked as if the data was never split. That's what sold me on GNU Stow's ability to produce the merged directories I'll need to pull off more fun stuff with this hierarchy.

The careful reader already knows there's a caveat coming: synchronization. How does one keep them synced? The first solution that came to me is a cronjob that runs stow with the --restow (-R) flag:

*/15 * * * * stow -d /usr/local/stow -t /home/zlg/music -R music_others music_purchased

You would repeat this for each stow target you want to work with. Doing it every fifteen minutes can be a bit too often for your use case. I used a five minute interval during testing and it didn't seem to tax my system.

Tidying Up

With these changes and automation in place, I needed to go correct a bunch of my existing symlinks to suit the new hierarchy. It was mostly manual work, but with a few find pipelines it was trivial to find the broken links. if I were to repeat this, I would instead have a stow package dir containing symlinks to other files, but with the filenames that I need in $HOME for everything to hook up nicely. I'll need to test that case a little more, because the links would need to be relative to the target directory instead of the stow package directory. If I pull this off, I can name it 'integration' or 'homedir' or something, and my dotfile management will be a solved problem.

Conclusion

I'd be remiss if I didn't share the structure that I ended up with! I did the vast majority of brainstorming in a neat piece of software called TreeSheets. It was the first time I'd used this software, but I enjoyed it quite a bit and will definitely use it for my next big brainstorming session. Hierarchical spreadsheets are a wonderful idea!

Click to download the original TreeSheets file (.cts)

I'm quite pleased to have mostly solved my organizational stress problem. It's relieving to know things are filed away in a sane and predictable manner. I'm also happy to have another tool at my disposal that will save me time.

Thanks for reading,

-z

(mkbak is) A bash script that's a thin wrapper around rsync. It takes a single rsync filter list (to allow both including and excluding) and backs files up to a given location, preserving their permissions and mtime, etc. Maybe I will release it some time. ↩
Due to my living situation, I no longer run the desktop all day, to save on electricity costs. Thus, I needed some spare "scratch" space. ↩

Zettelwhat?

2020-06-06T01:03:00-07:00

I checked out the Zettelkasten method, which appears to be German for something like a box of (index) cards.

The website for the method appears to be a marketing site, almost. It's not simply and plainly spelled out for you what the method really is, and it lacks clear, real-world steps and examples to get the most out of the method. I dug a little and finally found the gist of it, being:

Write a note on a card for a single topic
Give it a permanent ID
Link to this note from other notes, via this ID
Maintain links via a master link list
Use the link lists to form associations between knowledge
(Multiple link lists may be used to map your knowledge in different ways)

The grand goal is to have a well of knowledge that you can, eventually, begin to interact with and explore the limits of your own documented knowledge, which I think is an interesting distinction to make. One of the reasons we write notes is to externalize information in a form we can recall later, to offset the cognitive load of short-term memory. Then we only need to remember to come back to our note(s) to be reminded of what we already know.

So as a habit and idea, it's nice. But when you think about it, this is exactly how a wiki works, minus the "write once, link forever" aspect of Zettelkasten. It's also how you would more or less add tags to data in a well-normalized database. You would implement it using a lookup table, like "genre_lookup" below:

      t_game                                          t_genre

+----+------------+                             +----+-----------+
| id | title      |                             | id | genre     |
+----+------------+                             +----+-----------+
| 3  | Beep Boop  |         genre_lookup        | 8  | Action    |
+----+------------+                             | 9  | Adventure |
  |                       +------+------+       +----+-----------+
  |                       | t_id | g_id |         |
  |                       +------+------+         |
  '---------------------> |  3   |  8   | <-------'
                          |  3   |  9   |
                          +------+------+

This lookup table means 'Beep Boop' is Action/Adventure. (We should add Sci-Fi too, but I digress. ;) If this were a Zettelkasten, it would be called "Game Genres" and link games to their respective genres. It would just be in a less verbose form, like:

game3 -> genre8, genre9

The exact field names aren't important, it's just an illustration. One could have a convention for fields so you can fit more on one note. Something like "capital G for game, lowercase g for genre" would be enough:

G3 -> g8, g9

Then if you really cared you'd look up g8 and g9 to find Action and Adventure. A more pragmatic person would create a note that has the list of genres on it, and update it (with permaIDs) as they go along in similar fashion. That way you only look up one "note" to find all genres. Then our link card could be:

G3 -> 8, 9
(see: genre_table)

Then the 'genre_table' note would have a simple numeric list of genre names.

This little mental exercise helped me conclude that Zettelkasten as a way of approaching one's knowledge is interesting and maybe useful, but the tech is more or less already implementable in traditional forms, like wikis, static HTML, vim-wiki and org-mode, etc. Even Gophermaps can be "zettel". I think their site could benefit from pragmatic, real-world examples of using this stuff and a focus on how to apply the method to various media. There are apps for it, but they all seem to have various warts, so it would be more wise to use extant technology to try the method.

Zettelkasten strikes me as a mindset or an idea rather than a method, because it's not clear how long a zettel should be, if they should be edited or changed over time, if links can be deleted or modified as you refine the knowledge base, etc. An effective method has answers for these finer points, presented in a form that's digestable in a single webpage instead of a series of blog posts waxing philosophical about a method that amounts to tagging cards.

Neat idea, but it talks itself up way too much. Just my two cents.

-z