As my time in Chicago ends (for real! I’ve shipped my bike to Portland!), I’ve been feeling a nostalgic longing to be in all of the places I’ve ever known. I want to be wading through a cornfield near the Susquehanna River, tripping over bricks on Locust Walk, admiring the Hudson from Riverside Park, and struggling to write blog posts at DBC Chicago, all at once, all the time.

Whenever I feel this way, I’m reminded of the Tralfamadorians from Slaughterhouse Five (yes, I used to be a huge Vonnegut fan, before I swore off dystopian fiction) – they’re an alien race that express puzzlement at the concept of time because they have complete awareness of all events in the past and future. It’s fascinating to try to apply this to a human life. We aren’t only who we are currently, but rather a collection of our past and possibly future experiences. Maybe we’re even a collection of experiences of everyone close to us, everyone we’ve met, everyone who exists.

I’m also reminded of how I’ve loved being able to see stars on the clear nights in this city. Stars are important, you know? Normally my thoughts are earthly and immediate, but noticing stars throws them farther – to the realization that we’re small in the universe, but we’re also part of something wonderful and much larger than we are. It’s a reminder that there’s beauty everywhere, even in (maybe especially in) darkness and calm, and that this spectacular view is accessible! To us! Merely by pausing and tilting our gaze upwards. How incredible is that? (How absurdly fortunate it is, that we have eyes, that we exist, and the universe exists too…)

It’s important to look often to things that are pure and inspiring, even when distractions are plentiful and there’s pressure to focus on the mundane present. It’s easy to stay wrapped in familiar things like work, relationships, buying things, eating, and forget that our reality is so much larger than what we can comprehend in any given day. But if we have the time to look outside ourselves, we’re able to measure our current struggles with perspective and envelop others in our reality.

Today’s been tough for me: I love it here and I’m devastated to be leaving, I still can’t get my nginx server running consistently (impostor syndrome, anyone? I’ll write about this once I’ve figured it out), and I can’t stand cleaning. I’m terrified to be moving, but I trust it’ll be okay: there are incredible things in my future (in all of our futures!) that the human parts of me don’t know yet.

To Chicago – thanks for the friends, the popcorn, the new beginning, the stars.

This is by no means advice on how to parse PDFs in Ruby, just a summary of what I did over the weekend and an open invitation for other people to tell me about their approaches.

I was working on extracting text and images from the US Figure Skating Association rulebook, specifically from pages with moves in the field patterns, like this one:

Basic Consecutive Edges - USFSA Rulebook 2014-15

 

There were about 70 pages with a similar structure, and I wanted four things:

  • the test level from the title (Adult Pre-Bronze)
  • the name of the pattern (Basic Consecutive Edges)
  • the description (the rest of the text)
  • the diagram

Ruby Gem – pdf-reader

I started out using this gem, which takes a file and creates a PDF Reader object. This object then has a page method that returns individual page objects, which then have other elements. Using this gem, I was able to iterate over the pages of interest and grab each line of text.

Much of this is painfully manual and depends on how standardized each of these pages are – for example, I hard code that the title is always the first line of the page. I suppose this is commonly the case with scraping/parsing tasks. So this isn’t great, but since this rulebook gets updated at most once a year (and most of the descriptions will stay the same year to year), I figure it’s tolerable.

There’s some weirdness that comes out of this that I haven’t started refining yet. Here’s the text I extracted:

• Forward outside edges • Forward inside edges • Backward outside edges • Backward inside edges Starting from a standing position, the skater will perform four to six half circles, alternating feet, using an axis line such as a hockey line. The skater may start each set on either foot, but they must be skated in▯ the order listed. x: 0.3914 Focus: Edge quality

What’s with the ▯? And the x: 0.3914? (some positioning issue?) Probably not meant for humans, so this is something else to work on.

Overall this was very helpful for text, but I wasn’t able to get the diagram with this gem. This example code is part of the gem files, but I wasn’t able to pull xobjects from my pages to extract images in the same way. I was concerned that the labels within the image (e.g. LBI, LBO) wouldn’t be pulled properly anyway, since the labels were pulled out with the text method – it seemed they were overlaid instead of being part of the image.

Another Ruby Gem – docsplit

I don’t know much about the full capabilities of this gem, since I used it purely to split the Rulebook into 485 image files (one for each page). This was pretty easy, and I’m sure I could have done this with many other PDF tools with user interfaces, but why not?

One More Ruby Gem – RMagick (and ImageMagick)

At this point, I just needed to crop each PDF page image down to the diagram. Again, I’m sure I could have done this pretty easily with a some sort of graphical user interface, especially if I was okay with setting the same cropping dimensions for each page (which would probably be good enough). I’m glad I didn’t, though, because both of these tools were interesting to learn, and I can see many more instances where I might use them in the future.

ImageMagick is a command line software to manipulate images, and RMagick is a Ruby interface for ImageMagick. This allowed me to create a Ruby image object from each page, and then look at individual pixels in the image. These were the steps I took to extract a cropped diagram from each page:

  1. I sampled the pixel colors of one full page image into a hash (color: frequency pairs) to isolate the grey background common to all of the diagrams. Fortunately they were a uniform grey color, and I used the hex code as a variable in later steps.
  2. I wrote methods to sample pixel colors over a grid on each page so that I wasn’t checking the color of two million pixels for every image – I ended up checking one full row/column of pixels every 100 pixels.
  3. I wrote methods to find the spot of the left-most, top-most, etc. color change – since the diagrams have rounded edges and occasionally white line breaks, this required looking through the color results from the previous grid process and finding the most extreme row/column (i.e. the one where color change happened closest to the edge of the page).
  4. Using the excerpt method for image objects, I was able to take the x and y limits of the color changes and use the values to crop each page to a custom size, with some padding around the grey diagram.
  5. Finally, I was able to use the write method to save each cropped image to my Rails assets, and then use these cropped diagrams in my application.

My code for this DiagramSelector class isn’t very concise, but it’s here on github if you want to take a look. There’s also some relevant work in this seeds file (including what I posted above for the text extraction part).

Working with RMagick was definitely the most exciting part of this process – these kinds of tools can be used for Instagram-like filtering, automated laboratory data collection, all sorts of things! Image processing is definitely something I want to look into further in the future. One project at a time, though.

Box-herding (noun) – the act of pushing elements around a web page, particularly through adjusting position by pixels in CSS; typically accompanied by frustration as boxes gain sentience and make unpredictable attempts to escape envisioned layouts.

For a long time I conflated page styling and CSS with box-herding and considered it to be a superficial pursuit, clearly less important than back-end functionality. I’ve made peace with CSS as I’ve slowly come to see it as an opportunity to build a structure rather than move things around.

So here are my recommendations for learning to not hate CSS. (I use “my” loosely because most of these recommendations are from instructors at DBC and Ryan Bahniuk.)

1. Reconsider Purpose – Is styling superficial? Does superficial == not meaningful?

Yes, semantically, styling is superficial – it affects how things look on the surface of your application. However, good styling integrates with the deeper logic of an application and presents information in ways that make sense of intended functionality. So in most cases, styling isn’t purely superficial.

And even if styling is superficial and divorced from functionality, it’s wrong to believe that it’s not meaningful, as long as humans are viewing it. As consumers of information, it’s reasonable and efficient to prefer pages that are well-designed and well-styled, since they’re more memorable and require less time to understand. Styling is a way to communicate with intent – good for web development, good for life!

2. Separate vision from execution.

Most people start using CSS just to make something “look better,” and haphazardly throw in some colors and margins and hope that the ultimate result is slightly better than a black and white page. This gets draining because it requires making minor decisions all the time (“is this box far enough to the left? should I use a vertical menu instead?”).

I’ve found that this decision fatigue is vastly alleviated by separating the designing of a page from actually writing the markup and CSS. Start with a piece of paper, a wireframing site (I’ve used a few from this list), photoshop, whatever, and draw out a mockup first. Having a final goal forces you to think about how elements of a page interact with each other and the most logical way to layer things together, rather than pushing things around as independent entities and getting annoyed when they bump into each other.

3. Be uncompromising.

While coding, don’t give up on your designs – use intention to train your skills, and don’t allow skills to dictate your intentions. Stray from your initial vision only if something else makes more sense for communicating what you want, but not because you can’t figure out how to align your divs or extra pixels are sprouting out of nowhere. There’s usually a way to do something, and you’ll eventually feel much more like a builder if you’re executing designs faithfully (with googling along the way) than if you take shortcuts when things get tough.

4. Probe details.

Sometimes styling devolves into a tangled mess of interrelated elements, and it becomes impossible to extract why elements are shifting in unexpected (and undesirable) ways. At this point, tools to look at elements individually are helpful.

Grow accustomed to using inspect elements in your browser. It’s the best way to visually relate elements to markup and applied styles, and it provides numerical information on sizing of margins, padding, borders, etc. Click on specific elements and experiment with additional styles to get you closer to your design, then add those to your stylesheet.

I also like using codepen and other sandboxes to test out individual principles (e.g. how to align three divs adding up to 100% width in a horizontal row). Once you’ve solved a specific problem at the lowest level of complexity in a sandbox, it is much easier to apply your solution to a full project.

5. Read technical CSS stuff.

At some point after going through basic tutorials, I felt like I had the tools to [poorly] code up most things I could envision. This is deceptive though – at this point, designs tend to be fragile and difficult to modify. And while using stackoverflow for solving specific problems is a practical way to make progress, it isn’t usually best for developing a deep comprehension of styling logic.

There’s a lot of technical CSS writing out there, and investing time in reading more narrative-style articles can be better for uncovering the logic and understanding best practices. The W3 standard recommendations for CSS (latest revision) are useful for understanding how elements are designed and what styles they take on by default (e.g. inline vs block, padding, margin). I was recently reading over this list of topics from Smashing magazine, which publishes many detailed articles on CSS principles and how to use them well (a google search of “CSS design articles” turns up more information like this). The more technically you understand things, the less you’ll be tempted to put random numbers into your styles to box-herd. And the closer this will feel to “real” programming.

6. Seek inspiration!

Talk to people who enjoy styling pages – the ones who salivate at innovative, responsive web pages and grumble at single-pixel misalignments. Pair with them to see how they make styling choices and ask them for advice when you’re making decisions or run into problems.

Also look at examples of good or interesting CSS on the internet. Codepen features “picked pens” on their front page, and a.singlediv.com features CSS artwork made from manipulations of single divs. These aren’t exactly practical, but they’re fun to look at and a great reminder of what’s possible. More practically, inspect elements on web pages you enjoy. Look through the source code for CSS frameworks like Bootstrap: modular, flexible CSS tends to be a good aspiration. Also, if you decide you really still hate CSS, you’ll know how to use Bootstrap.