R: How to fix column names containing spaces

A basic rule of R is to avoid naming data-frame columns using names that contain spaces. R will accept a name containing spaces, but the spaces then make it impossible to reference the object in a function. Neither single or double quotes can work around this problem, and other data structures also share this limitation. (Note that commas also cause similar problems, as do many special characters.)

> select(x, Date, Depth, Total Phosphorus)
Error: unexpected symbol in "select(x, Date, Depth, Total Phosphorus"
> select(x, Date, Depth, "Total Phosphorus")
Error: All select() inputs must resolve to integer column positions.
The following do not:
*  "Total Phosphorus"

When I’m creating my own column names it’s easy to avoid the issue, but often I’m working with data that comes from other sources — and contains spaces in problematic places.

At the moment I’m working with lab data where every water-quality parameter (nitrogen, phosphorus, copper, temperature, etc.) is its own line in the dataset, with one column for the parameter name and another column for the value — often called “long” or “molten” format. The names-with-spaces problem comes when I spread out the data into “wide” format, where each row is a complete sampling date and each parameter becomes its own column.

Many of the parameters have spaces (and sometimes commas) in the name the lab uses, such as “Ammonia Nitrogen” or “Copper, Dissolved”. When I use the spread() function (from the “tidyr” package), these become column names containing spaces and commas. After trying a few approaches, I found a simple, elegant way to do it in a single line using the “stringr” package.

To fix a dataframe (or any other named structure) “x” that already has names containing spaces and/or commas:

names(x)<-str_replace_all(names(x), c(" " = "." , "," = "" ))

This replaces spaces with periods, and commas with nothing (to avoid creating double periods like “Copper..Dissolved”). Spaces and commas are the only two problematic characters I have in my dataset; if you have others, add them to the str_replace_all() call. The result from a names() call is a vector of strings, which the stringr functions can manipulate easily before passing it back through the names() function.

Another approach is to remove the spaces from the long/molten dataset before spreading it out. It’s very similar code, only on a column of parameter names:

y$Parameter<-str_replace_all(y$Parameter, c(" " = "." , "," = "" ))

 

UPDATE: I’ve discovered that tibbles — the tidyverse version of the dataframe — can handle names with spaces and other special characters. To refer to those non-standard names in functions, surround the name with backticks (on my keyboard, the backtick is on the  ~  key):

`​Copper, Dissolved`

 

Even though I’m working primarily with tibbles now, I’m going to keep renaming my columns so they’re easier to work with as I go through the rest of the data analysis. I prefer not to have to type the backticks all the time.

Leave a comment

Filed under R

When hatred isn’t urgent

I don’t think that Trump’s platform of hatred and bigotry got him elected. I think that too many decent people were willing to vote for him despite his racism, sexism and all the rest — because they thought other things were more important. The hatred simply didn’t seem like an urgent problem.

But when racism means that I could get shot by the police, it threatens my basic ability to survive. When sexism keeps me from getting a job, it threatens my basic ability to survive. And when something threatens my basic ability to survive, it’s urgent.

Too many of us with privilege don’t see hatred as urgent. We know racism and sexism are sorta problematic, sure  — but other things are more pressing. We’ll deal with them later, when we’ve got time and energy. If we remember to. If it doesn’t get in the way.

We need to change. 

Continue reading

Leave a comment

Filed under Uncategorized

You don’t know what’s in your cleaning products

8115464937_bc1a808bfb_zWhat’s in your household cleaning products? Even if you’ve read the label, there’s probably still a lot left off. No law requires a complete list of ingredients on the labels of household cleaning products, unlike food or cosmetics. And what they’re not telling you about could still hurt you.

Continue reading

Leave a comment

Filed under Uncategorized

Wading through the red tape to get a government job

Now that I work for the county, I’m getting questions about how to apply for jobs here. Government job applications play by different rules than the rest of the world, so most of the advice you find out there on resumes and cover letters don’t quite apply. I’d like to expand this further (with collaborative input) to cover the whole job application process, but here are some tips for now:

The first round of screening is a “minimum qualifications” round where anyone who doesn’t meet each and every one of the listed qualifications will get excluded. It can be brutal.

To help pass the qualifications screening, HR recommends copying the list of qualifications verbatim from the job posting and answering each one. I do this in the cover letter. I write a normal one-page cover letter, and then on the next page(s) I address the qualifications.

If you’re applying for a job that requires X years of experience, address that with a table that lists each position title, years in the position, and totals them up. Either include a statement that all jobs were full-time, or include an hours-per-week column and a full-time-equivalent column. Also, on your resume put the hours-per-week for each position. (It’s in the online application, but most people will use either the online or the paper copy, not both.)

For resumes, the usual “short and sweet” approach that works everywhere else is counterproductive when applying for government jobs. Your resume has to stand alone to prove that you’re well qualified for the job (if you’re hired, it actually gets submitted to the state for oversight). That means that it needs a fair bit of detail to show that you do indeed have all the skills, knowledge, and experience you need for the job. Yes, your resume will get long in the process.

The advice I was given by HR when I was applying is to make sure that all the keywords in the job description and qualifications are addressed in your resume. You don’t necessarily need to use the exact words, but you do need to make it very clear – the first screener may be an HR person who doesn’t understand the field well enough to understand specific terms — and in any case the screener will be working bleary-eyed through a stack of ~150 applications.

Leave a comment

Filed under Uncategorized

Fertilizing streams to save salmon?

forest stream - Barka FabianovaIt’s not often I read a technical report that’s jaw-dropping, but this one sure was. It’s the Fraser Valley Stream Nutrient Enrichment effort, where they’ve been adding phosphorus fertilizer to salmon-bearing streams. Much of my work focuses on keeping excess phosphorus (and nitrogen) out of lakes and streams — and here’s a big effort dumping it directly in?!? What’s up with that? Continue reading

Leave a comment

Filed under water

Unleaded, please!

The last place I expected to find worrisome amounts of lead is in something designed to pipe water around the home — like a garden hose. But HealthyStoff.org (a project of the Michigan-based Ecology Center) tested 21 hoses, and found lead, phthalates, BPA, and other compounds of concern, including chemicals that have been banned from children’s toys. In a test where they left a hose in the sun for two days and then tested the water, BPA and phthalates far exceeded federal drinking-water standards.

Vilsekogen - garden with hoseI had never given it much thought — sure, garden hoses aren’t really built for drinking out of, but they’re used all the time for water we end up ingesting, whether that’s watering the garden, playing in a sprinkler, or filling up a water jug to take camping. If you had asked me yesterday, I’d have guessed that maybe they’re not officially food-safe, but they must meet some kind of standard, right?

Nope. Continue reading

1 Comment

Filed under Uncategorized

Are you a lawn lemming?

Do you fertilize your lawn? Pull weeds? Water?

More importantly — do you know why you do these things?

The Washington Post recently ran an article highlighting the role of social pressure in people’s decisions about fertilizing their lawns. One of the studies they highlighted was just published by some of my urban-ecology sociology colleagues at the University of Minnesota (Nick Martini and Kristen Nelson): Continue reading

Leave a comment

Filed under lawn

Compost conjectures

creative commons license, by Sarah Gilbert

How much food waste do Minnesotans compost at home? This is one of those questions where we really don’t know the answer. Backyard composting is a purely private affair, one that doesn’t pass through any sort of permitting or reporting process. So what’s an ecologist to do? I often find myself in a position like this, where nobody has measured the thing I want to know — but I can piece together enough information to make a decent estimate. It’s a fascinating process, so let’s pull back the curtain and take a look…

Continue reading

Leave a comment

Filed under compost, Minnesota

Don’t panic!

We were up on Mount Rainier last weekend, doing some snowshoeing up at Paradise and hiking a bit lower down. There is a lot less snow than usual, both a thinner snowpack and a higher snowline. And around here, snow = drinking water…. So back home today I took a look into the SNOTEL data and it’s pretty bleak:

SNOTEL snowpack mapWow, the central Cascades have only nine percent as much snow as “normal” for this time of year. Last winter, when drought seemed to be on everyone’s radar, a wet February brought conditions back up to normal. But we’ve had a warmer winter, and that hasn’t happened this year. So why is nobody panicking?

The WA Department of Ecology’s current drought watch is a bit confusing. They’ve requested drought-emergency funds be set aside, the forecasts predict a warm and dry spring that will exacerbate the problem, and yet they say “Ecology and [the Water Supply Availability Committee] at this time are not anticipating widespread water shortages in Washington in 2015.”

Farmers will know more in the next couple of weeks as determinations are made regarding irrigation uses. Here in the city, though, we’re actually in really good shape. There’s been a decent amount of precipitation this winter — just mostly as rain, not snow — and Seattle’s main municipal reservoirs are filling nicely (I haven’t found quantitative data yet). Unlike some parts of the West where the snowpack itself is the primary reservoir storing water, Seattle has dam-and-lake reservoirs so we’re not directly dependent on snowpack.

So the bottom line is: Don’t panic! Yes, we have very little snow — but plenty of drinking water.

Leave a comment

Filed under Seattle, water

The costs of being green

It’s not easy being green — and it turns out it’s not cheap, either. This certainly isn’t what I expected to find when I started looking into the costs of waste disposal in Seattle.

Trash and organics picked up throughout the city are first taken to one of two transfer stations, North (in the Fremont neighborhood) or South (in the South Park neighborhood). The North Station is temporarily closed and being reconstructed, so as of this writing everything is going via the South Transfer Station. From there, Seattle’s trash travels over 300 miles, by train, to the Columbia Ridge Landfill in Eastern Oregon (view map). Organics are split between two composting companies, Lenz in Stanwood, WA (60mi from the South Transfer Station, or 50mi from the North Transfer Station; view map), and PacifiClean outside Cle Elum (98mi from either transfer station; view map). Recycling is taken directly to Rabanco, in SODO (the neighborhood SOuth of DOwntown), which for much of the city is a shorter haul than the South Transfer Station. Continue reading

Leave a comment

Filed under biosolids, compost, garbage, recycling, Seattle