Google, Robots…iPhone?

OK two of my favourite companies have failed me recently: Google & Apple. Or more specifically: Google’s robots.txt protocol support and Apple’s iPhone via O2’s service.

The iPhone’s over there, mocking me with a ‘No Service’ signal. But it’s not no service. Or it is no service. The forums are confused.

My bet is the forums are all a little right. I’m going to let it spend the night checking away. I’ll do a couple of resets in the morning (I’ve firmware reset twice this evening) and we’ll see how we go.

One thing I’m definitely leaving to last is calling the O2 support line: that’s one thing everyone agrees is shocking. My tolerance for holding on a phone is low. Especially when you have no indication where you are in the queue. I mean: not even a number?

I guess that’s Apple/O2. Google?

Everyone knows the robots protocol. It’s a fundamental web protocol that everyone supports. Google points people their way, and have always complied with their standards. Of course they do. Everybody does (everybody ethical, that is).

Even when you try to be a bit flash and use the REGEX functions to trim back the length of the robots - which is not part of the protocol, Google supports it. Hell, they even give you a few tips to get your REGEX right.

Except they don’t support their own guidelines. The oldschool SEOs reading this (hello) will be going through the same emotions I went through: a wry sense of deja-vu and a geeky intensity of nailing down the flaw.

In this case it’s simple. A quick phone call from Matt would likely fix it.

Essentially: Google’s robots’s wildcard matching is broke.

Wildcards disallows to exclude filenames plus a query parameter aren’t functioning as disallows, and are even being flagged in Google’s Webmaster Console as not functioning.

For example:
User Agent: *
Disallow: *brokeniphone.php?

or even Google’s own example:

User Agent: *
Disallow: /*?

Return the below:

It’s a pain for all the sites I’ve set up wildcard rules for as I’ve got a long debugging session looming as a result. I’m no Google hater - far from it! - but this used to be A-OK, it’s documented in Google’s Webmaster Pages, and it’s not functioning.

Please, Matt: give us webmasters an early Easter Egg.

<disclaimer>Google used to support this wildcard matching - at least up until a few weeks ago. They seem to have turned a dial somewhere since then.</disclaimer>