When your app composes tweets: Dealing with metadata

For those who don’t know, Twitter converts every URL to its own “t.co” shortener URL. So no matter how short or long your original URL is, the t.co URL will end up as a fixed character length, and that character length does count towards the 140 limit.

Any sane Twitter client will hide this complexity from end-users. The word count algorithm will be smart enough to take this into account show the remaining characters.

But as a coder, you need to incorporate that logic yourself.

You should also know that Twitter’s API won’t automatically truncate a tweet, so if your app tries to send a long one, Twitter will return an error. So your tweet-posting app will need to truncate the tweet to 140 characters.

So I was coding up an auto-tweet setting, which requires you to estimate the length of a tweet. The code looks like:

  1. TWEET_LENGTH = 140
  2. TWITTER_URL_LENGTH = 19 // !!Danger - read on!!
  3.  
  4. def compose_message(episode)
  5.    hashtag = '#nowplaying'
  6.    url_and_hashtag_suffix = " #{episode.url} #nowplaying"
  7.    max_title_length = TWEET_LENGTH - (1 + TWITTER_URL_LENGTH + 1 + hashtag.length)
  8.    "#{truncate episode.title, max_title_length}#{url_and_hashtag_suffix}"
  9. end

And then, with a long title, it failed. Can you guess why?

The answer is because I apparently went to sleep for three years, and when I woke up, the world had composed hundreds of billions of tweets. Many of them include URLs, which means the t.co length has crept up to 22 characters – 23 for SSL URLs – rising at about 1 character a year. Yes, if your tweet has a link in it, you now have to be 2.5% more concise in describing the link (that’s 3/(140 – 19)).

Thankfully, there’s an API for this:

So your code could periodically crawl the config API and aggressively-cache the result. Or alternatively, have your build script download it to your code base at compile-time, if it hasn’t seen an update for a while.

I haven’t checked in detail, but there are probably some open-source Twitter packages (gems, NPM modules etc) that include this config data and keep it up to date.

Note this also affects images and video – the above config URL also provides the length of a media item.

Someone Different: Encouraging Twitter Serendipity

I made a very raw cut of an idea that’s been percolating for a while. The idea is to randomly follow a few people for a month or so, then rotate to a few different people, and so on. This would let you immerse yourself in a community for a little while, with the aim of busting out of the filter bubble, learning something new, and maybe making a new friend or two.

Demo: You can try a proof-of-concept here (don’t be put off by the permissions Twitter asks for; it won’t tweet on your behalf, the permission is unfortunately necessary to manipulate lists). It’s raw, but will create a real list for you and set its membership. It works now by simply switching to one of several manually-curated lists, which I just rotate when you visit the app (/pages/welcome). If there’s interest, the plan is (a) to make the rotation happen automatically (b) generate the lists from something like Klout or PeerIndex, to ensure you’re following people in the same community.

Right now, the “someone different” people live only in your “someone-different” list, but I’m inclined to follow them in the main stream. As long as they stay in the “someone-different” list, it’s still easy to unfollow them from the main stream. And if you find you want to keep following someone, you just remove them from the “someone-different” list. (It’s all very ghetto here; I’m basically using the twitter list in lieu of maintaining a database.)

I recently noticed @jobsworth was also looking for something like this. His ideas are certainly not achieved in this v0.0.1, but hopefully it will get people thinking about what’s possible. Please let me know where you’d like to see this go.

The app’s built in Rails using the very nice twitter-login gem and hosted on heroku.

Google Plus: The Good and the Bad

I’m on Google+ and I’ve mostly found it pretty decent in the ~2.5 weeks I’ve been using it. (Also had an opportunity to do dogfood before it went live, but really, it’s a different world once it’s out there and being used for real, so that’s what this post is about.)

Thought I’d sum up my experiences to date. You’ll notice a lot of my comments compare it to Twitter, because after all the crowd has been similar so far. And honestly, I often find myself with something to write and having to decide whether to stick it on T, G, or both. The decision feels kind of arbitrary right now and will probably shift to G over time, unless Twitter gets its innovation seriously on …

The Good

Things to like about Google Plus …

  • The UI! I’ll call out the design first because it’s not been Google’s strong point in recent years. Yes, clean white pages were a breath of fresh air in 1997, but getting long in the tooth now, and Google Plus shows Google’s now getting serious about visual design, and beyond just statistical hue calibration.
  • Engagement. The overall experience leads to much higher engagement than I’ve seen with blogging, twitter, or anything else. For the big guys, you see literally dozens of comments in the first few minutes.
  • No size limit. You don’t have Twitter’s 140 character limit. Fortunately, Twitter taught us the benefits of brevity, so it’s very acceptable to write just a sentence or two. But you can still go all out with an entire paragraph and that’s fine too. And no more shoehorning txt,just 2fit the arbtrary char limit.
  • Comments. Compared to Twitter, I actually have all responses to my status in one place. On Twitter, there’s no UI to see all the responses to a tweet; you have to view them all individually. The comments also facilitate a conversation, whereas on Twitter, you’d end up with multiple tweets and no decent UI (save third parties perhaps) to view the whole conversation.
  • Mentions. Similar to Twitter @ mentions, but because of the comments, + mentions have more of a feel of being summoned into a conversation. It’s certainly easier to get the entire context of the conversation in which you’ve been mentioned.
  • Search. <rant> Seriously a shambles about Twitter is the inability to search through tweets. I can’t even search my own tweets!!! This was one of my initial motivations for using posterous, figuring I’d pipe everything through it. And same reasons others use identica, etc. The problem is you don’t get all the clients, e.g. Tweetdeck. So I ended up sending a lot of tweets which I can’t actually search through. They fall off Google and they fall off Twitter’s own search. It has constantly amazed me that a company in constant search for a revenue stream doesn’t have a full history of content available.</rant> Now, with Google Plus, there’s no search either, but at least advanced users have the option to search using site:plus.google.com. And in all likelihood, we can assume Plus will have search too. (Although Android search shows Google + Search is not a foregone conclusion of excellence…but Plus’s virtues to date promise good things.)
  • Hangouts. I think the concept of video hangouts is immense, as opposed to just standard VC, and will be a great asset for remote teams. (I know some teams already just keep an open skype session running all day.) However, I’m speaking out of my speculatum because I haven’t yet tried it, for reasons explained below.
  • Updates. The team is doing a great job of engaging with the community, listening to the feedback, and making updates. They’re doing a great job if the worst controversy is closing down accounts of sophisticated tech companies blatantly flouting the rules by setting up corporate accounts, complete with filling in their first name, surname, and gender.
  • Android App. A fine production.

The Bad

Things not to like about Google Plus, keeping in mind it’s still in “field testing mode” (“beta” is so web 2.0) …

  • Circle statuses aren’t public. The problem is I want my Twitter-like geeky status updates to be public, but not contaminate the streams of my non-geek friends and family. There’s no good solution for this right now. If I publish a geeky status update to my “webdev” people, they will get to see it, but it won’t be public. If I publish it to “Public”, everyone in my circles sees it. Right now, I’m opting for “Public”, and if others are doing likewise, their non-geek friends and family will freak out the first time they join Plus and run fast in the opposite direction.
  • No groups. A lot has been said about whether circles themselves are a good idea, given that people’s relationships are nuanced and dynamic. But what interests me more than circles is groups. I’m really hoping Google somehow merges Google Groups with Plus; private groups would be particularly powerful for workgroups. The cool thing about groups, like what Facebook has but few people use is: (a) compared to circles, the visibility model is easily understood and predictable – someone can see messages if they’re in the group and can’t see them if they’re outside the group…simples! (b) you don’t have 100 people all adding the other 99 to the same circle and all micromanaging their circles to stay in sync – the owner just adds everyone once…this saves a lot of hassle, it works better for mainstream users who can’t or won’t micromanage circles, and ends up with a single source of truth.
  • Comments UI. As I mentioned on + Comments UI works for the average Joe with a handful of comments. But for Scoble etc, they are reedeeculous! 100+ comments in an hour? With no threading. It’s a mess to read and removes much of the incentive to post anything. This stuff is write-only. Since comments are +1able, G+ should let the popular ones float to the top, or perhaps display them with the others being collapsed. There are some other good suggestions in the comments to the aforementioned + post.
  • Comments and Sharing. The problem with sharing is people will comment on the sharer, not the original. I think it makes sense for everyone to be commenting in the same space, or at least for some easy summary view. I think this is Google being initially biasing towards privacy measures to avoid adverse reaction, but it does lead to comments being even harder to follow.
  • Finding hangouts. I can’t. Maybe that will change now that PlusRoulette is here! Or not. I also hope Google adds the ability to record hangouts at some stage. Lightweight video podcast ftw!
  • No metadata. I get it. Statuses don’t have titles or keywords or anything else because people are lazy and sites that don’t realise that won’t go mainstream. So Google’s being smart here. But that said, for my own use case, there are times when I’d love the ability to tag a post with “announcement” or whatever, so I can later have a feed of those. Anyway, I guess one can work around it by including some hashtag-like token in the actual status. Just making that point, though I realise my own use case is not that which Google is targeting – it seems the lesson has been learnt from Wave et al that Googlers and assorted geeks shouldn’t always be the target audience.

Okay, that’s me done. I’m enjoying Plus and I’m hoping it spurs Twitter and others to build on the awesome. Are you using it? Any other goods and bads?

Update: conversation on Plus

List Of Tweets

ListOfTweets.com is a little twitter mashup I made to organise tweets. The tool lets you build up a list of tweets from search results, which you can convert to HTML or text. The list doesn’t empty unless you clear it, so you can keep running searches and adding to the list. You can easily rearrange tweets with drag-and-drop.

By the way, List Of Tweets is another example of a URL Trail. Thanks to the simplicity of the Twitter API and the elegance of JQuery, it was possible to build a raw version in a couple of hours. t is approaching zero. Only after tweeting about it and discovering people found it useful did I decide to flesh it out some more.

A future upgrade might involve saving the list at a unique URL. It would be nice if any of the pastebins provided some support for this via a JSON API (admittedly with relatively short URLs – only up to 2000 characters could be reliably posted). But I will probably build my own, or just start a new frame and post into a pastebin.

I originally wanted this as I’ve been thinking about podcasting again and wanted to pull out some highlights from my recent tweets, to act as a schedule, and something I could publish as a playlist at the end. However, a colleague suggested a far more useful scenario, which is to find tweets matching a keyword, hence I added the search keyword parameter, which I now use much more often than the tweeter parameter. Here’s an example of someone using it in their blog to archive tweets about a hashtag.

I’ve already let people know about the tool via Twitter, but it feels incomplete until I blog about it. Done!

Explaining the “Don’t Click” Clickjacking Tweetbomb

I just noticed some of my Twitter friends posting the following mysterious message:

“Don’t Click: http://tinyurl.com/amgzs6”

It turns out this is a Tweetbomb. If you go to that link and click on the button below, you end up tweeting the same thing:

… thus all your friends see it and some of them click on it and re-post it, and so on, thus propagating the message across the entire Twittersphere.

TinyURL shut down the redirect quickly and Twitter has responded, but the same attack could arise unless measures are taken. Of which, more later.

So how does this hack work?

The hack is an example of clickjacking. (I’ve heard the term a lot, but only understood its meaning after the investigation of this tweetbomb described here.)

Firstly, it’s using an iframe to embed Twitter.com on the page. The iframe is essentially invisible, due to the CSS structure…more on that in a moment. But first, looking at the source of the iframe, we spot our mysterious message:

<iframe src="http://twitter.com/home?status=Don't Click: http://tinyurl.com/amgzs6" scrolling="no"></iframe>

Well, I didn’t know about this before, but it turns out you can set an initial value for the status field by passing in a “status” parameter on the URL. So here, it’s setting the status to out mysterious message. You can try it yourself by visiting http://twitter.com/home?status=Don’t Click: http://tinyurl.com/amgzs6. (It’s safe to do so – nothing will get submitted, honest!) For the lazy and untrusting ;), what you’ll see at that URL is something like this, assuming you’re logged in to Twitter:

However, their job is not done yet, because they also have to trick you into clicking the “update” button. This is where more cunning comes into play. Check out the CSS for the iframe – containing your Twitter.com page – and the fake button contained on this trickster website:

  1. iframe {
  2.             position: absolute;
  3.             width: 550px;
  4.             height: 228px;
  5.             top: -170px;
  6.             left: -400px;
  7.             z-index: 2;
  8.             opacity: 0;
  9.             filter: alpha(opacity=0);
  10.         }
  11.         button {
  12.             position: absolute;
  13.             top: 10px;
  14.             left: 10px;
  15.             z-index: 1;
  16.             width: 120px;
  17.         }

And the elements:

<iframe src="http://twitter.com/home?status=Don't Click: http://tinyurl.com/amgzs6" scrolling="no"></iframe>

<button>Don't Click</button>

We can see from the CSS z-indexes, the iframe is on top of the button. And we can see from the iframe’s opacity that it is completely invisible. Hmmm…so it’s on top, but completely invisible. If there was a button there, you wouldn’t be able to see it, but you would still be able to click on it. This is where the layout comes in (position, width, height, top, left). The iframe is positioned just right so the the (invisible) update button appears in the top-right of the web page at (10,10), right where the fake button is. So the Update button is directly above the fake button but invisible. The website shows a fake button and tempts you to click on it, but clicking on it causes the real Twitter button – inside the iframe – to fire. Thanks to the URL status hack, you have just submitted their planted message. And of course, you couldn’t see the message, because the iframe is hidden and because that part of the iframe is not present anyway due to the negative left and top settings. There is also some filler text on the trickster page, below the fake button – I guess this is in case some browsers expand the iframe height if nothing else is on the page.

The main way Twitter can prevent this kind of attack is to use the old “bust the iframe” idiom. As I explained in Cross-Domain Communication with IFrames, it’s possible for an iframe to access the URL of its container, and actually re-set that value. Thus the common pattern:

javascript

  1. if (top.location.href != self.location.href)
  2.      top.location.href = self.location.href;

If Twitter had that code on its web page, whenever it was loaded inside an iframe, the user would see the entire page clear and refresh to become the Twitter URL that was in the iframe. I actually think it’s a shame they would have to resort to something like that, because there are valid uses for iframes, as WebWait demonstrates. Any site that uses this pattern cannot bathe in the warm glow of WebWait :). But I can see why sites need to take this measure. (Update: Twitter has updated their code to bust out of the cache, just after I posted this!)

Twelebs: Following Twitter Celebs

This is a new site I made: Twelebs. Twelebs shows the tweets of 80-odd celebrities, updated each minute:

Each celebrity has a short bio-summary:

… and a more detailed bio, fashioned from their Wikipedia entry:

Celebs are organised into eight categories:

… with celebs from all categories shown in the sidebar

Feedback welcome as always.

A Contact Form

I recently got fed up of spam. Well, that happened a long time ago. What bothered me recently, enough to take action, was false positives, i.e. real email that fell into my spam folder. My primary email address, michael@mahemoff.com, gets about 3000 mails a day…so what goes in my spam folder, stays in my spam folder. There’s no way known I will ever fish it out.

People I know usually get through to me okay, but of course I also receive mails from people I don’t know – who found my mail from my blog, websites, writings, or offline meetings. For them, there’s a good chance of being marked as spam, especially if they use a subject line like “hello” which is a perfectly natural thing to do when you’re just checking in with someone you’ve met at a conference, for example, so you can keep each other’s contact details. (Incidentally, I still find it astonishing how many people maintain blogs – many of them meticulous, comprehensive, and full of goodness – but include zero contact info.)

So I finally bit the bullet and made a little PHP/Javascript contact form. It’s been working great for several months now. This kind of form is nothing new, of course. Mail forms were once ubiquitous in every book on CGI. But there were a few modern twists worth mentioning.

The form looks as follows:

The gnarly twists:

  • Captcha. Or, really, ReCaptcha. Actually, I prefer Captcha to ReCaptcha – I think the benefit of scanning books is great, but can’t justify the usability cost of having to enter two words instead of one (for no extra security benefit). Still, I wanted to try out ReCaptcha, and I also like the way its a popular, centralised, service, which means it should be better at detecting evil clients than a standalone library installation. The PHP plugin and instructions turned out to be incredibly easy to follow. A pleasant surprise that meant spam prevention took about 20 minutes to set up in total.
  • SMS option. Instead of email, you can send the message as SMS (text message to my phone). I need to improve the UI here a bit, ideally creating separate tabs for SMS and email. Anyway, the way this works is via everybody’s favourite micro-blogging service, Twitter. On the Twitter website, I set preferences so direct messages are routed to my phone as SMS messages. Ideally, I would then send a direct message to myself each time someone wants to SMS me. But unfortunately Twitter doesn’t let you send messages to yourself (silly!), so I created a secondary account (the venerable “mahemoff1”). Each time someone submits a SMS, I use the Twitter API to end a direct message from the secondary account to the primary account. This gets me a text message several seconds after the form is submitted. I also receive a copy in my Twitter account, as a bonus. (For all the talk about Twitter’s unreliability, its text messaging is lightning fast. It shows up on my phone long before it shows up on Air clients, which only poll once a minute or so. Ideally I’d upload a video if I had the time, to show how quick it is.)
  • A web chat facility. Everyone should have it, to avoid making people install AIM etc just to chat with you one time! I installed an open-source chat client so anyone can talk to me. In an ideal world, browsers would notify me when someone is trying to chat, but for now, it requires pre-arrangement. I would like to enhance the app so it uses the same SMS mechanism to tell me each time someone enters the chat room.