Superfast Development Presentation

Gave a presentation on Superfast Development – some tips and tricks I’ve learned over the years to quickly develop applications – at last week’s Front End Party. Here are the slides.

Tips for Building a Text-Based UI in Node

Over the past week or so, I’ve been working on building Straturgery, a text-based game, playable over telnet, written in node.js.

The game, being both text-based and only accesable over telnet has provided some interesting challenges. I can’t use ncurses (since telnet is text-only), and have to build my windows and interfaces by hand. Here are a few things I’ve learned.

Escape Codes

It’s a bit esoteric, but you find yourself using terminal escape codes often. It’s also the only way to do colors. Most of the documented VT-100 escape codes work and are outputted with the escape special character \x1b[. You can test for this character when charCodeAt() equals 27.

A very helpful escape is the clear screen code, \x1b[2J. Escapes are also used for printing unicode characters. For example, you can print one of the many helpful characters from Code Page 347, such as , with the escape \u2588. Escapes can also be used to position the cursor to make drawing the screen easier (more on this later).

Colors

Escape codes are also used for colors. Just about all terminals support 256 colors (though very few support blinking). To enter the 256 color pallete, use the escape \x1b[38;5 followed by the color number and m. x1b[38;5; is for the foreground and x1b[48;5; is for the background. More details are here.

You can do pretty cool stuff with just 256 colors. For example, to make a red gradient:

1
2
3
4
5
var gradient = "";
for(var i = 5; i > 0 ; i--) {
  var x = (i * 36) + 16;
  gradient += "\x1B[48;5;" + x + "m \x1B[0m";
}

To see the amazing stuff you can do with just colors and unicode characters, check out these explosions!

Hopefully anyone else who’s building any type of node-based telnet application.

Side Project: Watching All the Tech Job Openings in New Orleans

For a long time, I’ve been fascinated by websites that are simply a generated set of flat files. 9 times out of ten, the content you want to put up on the web doesn’t change enough to warrant a big back end database. Generating files (HTML and the like) up from and simply serving a bunch of static files is a great way to have a simple and fast website.

[[nolatechjobs.leesome.com](http://nolatechjobs.leesome.com)][nolatechjobs.leesome.com]

The problem: I’ve been looking for a job in New Orleans for the past month. Even though there are only three main job posting websites for New Orleans, it’s still a pain to check each one – I’d rather have one main web site from which I can see all the job postings.

The solution: NOLA Tech Jobs is simple website updated hourly that goes out and compiles a list of job postings from various job posting websites (Craigslist, Work Nola, and Nola.com). It’s powered by a cron script that runs hourly. The script does two things: scrapes the job posting websites and converts them in to Jekyll posts, then re-builds the Jekyll website.

I knew I’d be checking this site on my phone often, so I used Twitter Bootstrap to build out the user interface. The header image is responsive, thanks to some perl code I wrote.

An interesting thing happened when I was building the site. Even though I was powerless to refresh the page and trigger a refresh on the job list, I felt the need to. So I added a countdown timer, which somehow mitigated this urge to refresh the page (possibly by letting me know when it was “ok” to refresh the page). The countdown timer also had the added effect of being a moving element, making the page more interesting to look at.

This countdown box ended up taking much longer than I expected. Turns out, dealing with dates on the web is very difficult. I had to correct my server time with ntp, and carefully generate the two files that control the countdown.

The countdown logic works like this:

I debated briefly about doing localization, but decided to keep everything pinned to America/Chicago since that’s the timezone that New Orleans is in.

So, if you’re looking for a tech job in the New Orleans area, and are lazy like me, try NOLA Tech Jobs;

A Phone Without a Phone

The spectre of a Facebook phone is upon us. A new phone that won’t be tied to hardware; a phone that is simply collection of interconnected apps that combined serve the basic function of a phone. A phone created by Facebook, but hardware independent; like a spectre, the physical shape will differ depending on the encounter. This experience has been purchased piece by piece for billions of dollars, as much a phone without a phone as a Frankenstein monster made from mergers and acqui-hires that lays waiting on the operating table for lightning to strike. All that’s missing is a lighting bolt – the user interface – to energize these pieces into the experience of a phone, a phone by Facebook.

Yes, this is phone

The smartphone began as the pocket computer/PDA, a tool used by deal-making businessmen for the production of capital. An address book, a calendar, and a calculator; some of the standard features that have come from the corporate officer’s desk. But as the smartphone has transformed from a tool of capital production to a tool of social consumption over the past decade, other features, such as messaging, are now a required part of the smartphone idea.

Three kinds of messaging take place on a phone: SMS, e-mail, and app-to-app. Initially pay-per text, SMS messaging was introduced in 1992 and became popular late 90s, thanks to cell networks allowing cross-network texts. SMS messaging helped drive the market for “dumb” phones, positioning them as social, rather than business, tools.

Mobile e-mail took off in the late 90s as well, with BlackBerry – a phone seemingly built for e-mail – becoming the dominant smartphone of the petty bourgeoisie.

The third and most recent form of messaging, app-to-app messaging, is free, worldwide text and picture messaging as exemplified by BlackBerry Messenger (BBM). BlackBerry’s former hegemony of the market is thanks in no small part to BBM, enabling instant communication in places that had never had it before (due to a myriad of factors: cell phone network restrictions, poor internet, long distance, etc).

The smartphone’s transformation from a tool of capital production to a tool of social consumption (and paradoxically – a means of alienation), began when smartphone’s hardware changed to replace consumer electronic devices, causing massive market disruption. The iPhone helped eliminate the mass market for personal music players, people tired of carrying both an mp3 player and a cell phone bought the iPhone to carry just one device. Once smartphone cameras were introduced, digital camera makers were forced to adapt to a changing market or die.

Back in 2013, Silicon Valley was rife with rumors of a Facebook phone, culminating in the release of Facebook Home, a sort of skin for a Samsung or HTC Android phone. The project was not a success and Facebook Home was largely abandoned by the media. However, the application continued to be updated.

As with all tools, success depends on the user interface. While having a mediocre user interface will not break a product, having a bad one will (Windows 8). Conversely, having a stellar one will drive sales (iPhone).

Facebook Home, I believe, has become sort of a skunk-works for exploring what a totally unified user interface by Facebook would be like. For example, an interface by Facebook has circles around people’s heads:

Facebook Home Chat Heads (4/2013) [[Business Insider](http://www.businessinsider.com/facebook-home-photos-2013-4?op=1)]Facebook Home Chat Heads (4/2013) [Business Insider] Update to Facebook Messenger (10/2013) [[The Verge](http://www.theverge.com/2013/10/29/5041976/facebook-revamps-messenger-for-ios-and-android)]Update to Facebook Messenger (10/2013) [The Verge] Update to Instagram (3/2014) [[The Verge](http://www.theverge.com/2014/3/11/5496350/instagram-for-android-5-1-screenshots)]Update to Instagram (3/2014) [The Verge]

Maybe in the future, WhatsApp will have some stickers as part of the interface by Facebook:

Stickers in Facebook Messenger [[Endgadget](http://www.engadget.com/2013/04/24/facebook-messenger-android-stickers/)]Stickers in Facebook Messenger [Endgadget]

This is where the spectre of a Facebook phone, a grotesque techno-monster of interconnected apps that owns all methods of communication, comes in. Facebook has all the pieces to build their own phone (without a phone):

A caveat: T-Mobile Bobsled seems to be a one-off event, almost like a proof-of-concept. Facebook Events is not being actively developed in the skunk works of Facebook Home. I’m also not sure if owning a web browser is going to beneficial (in the long run) to the success of a future Facebook phone.

Let’s have a thought experiment: Facebook capitalizes on this idea – a phone without a phone – to release an app that can do everything a smartphone does, a Facebook Home that works in the way Silicon Valley first imagined, yet at the same time makes your phone into essentially a dumb terminal for accessing Facebook-owned technology. The app is built to run on any hardware and priced similar to Kindle Special Offers; the hardware free or cheap, but the end user would be shown ads. You would still have a smartphone, you would just do everything through Facebook.

In exchange for providing a cheap phone, Facebook would use this device to dominate location-based advertising, gathering data to inject ads into daily life.

So the spectre of a Facebook phone, a phone created by the world’s phonebook, is beginning to appear before us. This spectre can either become real and lead a new frontier for growth, or this moment will be remembered as a missed opportunity twenty years down the road.

Discuss on Hacker News

Scripting Out UX Changes to the Downloads Folder

The Downloads folder is pretty much a standard feature on any modern OS. I remember my first experience with it was on Mac OSX Tiger (10.4). It became a very handy and frequently visited folder; before I was just saving all my downloads to the desktop for easy access.

Now that every browser defaults to saving files in the downloads folder(as they should), I find my downloads folder getting way to cluttered. After a few weeks, I end up with a folder filled with hundreds of files (and folders from unzipping applications), and it becomes a pain to find the file you just downloaded. Plus the folder looks very cluttered when you open it.

Being a pack-rat, I rarely delete things from my Downloads folder. I probably should, but instead I wrote a simple bash script, tied to a chron job, that will re-organize my Downloads folder and give me a fresh “empty” one each day. I call it Empty Downloads.

Basically, the script grabs all the files you’ve downloaded that day and moves them into a backup/ directory with the current date. So you go from this:

1
2
3
4
5
  Downloads/
  ├── bar.1
  ├── bar.2
  ├── bar.3
  └── foo

To this:

1
2
3
4
5
6
7
  Downloads/
  └── backup
      └── 02-18
        ├── bar.1
        ├── bar.2
        ├── bar.3
        └── foo

The new files you’ve downloaded today go in the root of the downloads directory and are not moved until the chron job runs.

When automating this script, make sure you set the run time in the cron job to be at the end of the day and not hour 0 since that will run the next day and make for some confusing dates in your backups folder. I.e.

59 23 * * * ~/src/EmptyDownloads/script.sh

Enjoy!

Source on Github

Sorry My Rent Is Late, I’m Using Dwolla

Update: Dwolla contacted me soon after this post and put out an update to this issue the next week. Shout out to Dwolla!

I’ve been using Dwolla for the past few months to pay rent on my co-working space and I feel I need a space to vent my frustrations. If their transaction fee wasn’t so low (25 cents), I would switch to a more familiar payment processor since their user experience is so poor.

Dwolla’s central promise of being able to send money to anyone, anywhere is appealing. However, the execution is a little bit off. Often times, I don’t need to send money to anyone, anywhere. I just need to send money to a select group of people.

Here is a screenshot of my Dwolla transaction history, showing payments made for my co-working space.

My Dwolla Account Summary

What’s wrong with this picture?

The phone numbers are all the same, but for some reason Dwolla doesn’t recognize that the 812 number belongs to Kristen (to whom I pay my rent). This is unnerving since Dwolla can’t seem to keep track of people’s names and associated phone numbers, I have no idea if they payment went out or not. Dwolla has already lost my trust by missing this important part of user experience, so I don’t have faith the rest of the service works. (Spoiler alert: it doesn’t; we’re still trying to sort out how payments work on this thing)

Why do the other payments show Kristen’s name and not this most recent one? I have no idea.

Even more unnerving is the drop-down menu when you go to pay someone.

The Dwolla Send Money Screen

As mentioned in the image, none of these are the right Kristen! It seems logical that the most recent people I paid would float to the top of the autocomplete list, or at least be in the list at all, but I guess it’s more important for Dwolla to show me random users that are also named Kristen in the off-chance I want to send them $200 for rent in the co-working space.

So here are my requests for a better Dwolla user experience:

So now it’s July 2nd, the money sent to Kristen’s phone number has still not arrived (nor has it shown up in her payment history). We’re now trying to see if Dwolla will recognize the e-mail address associated with Kristen’s account.

Thanks Dwolla, for not providing an easy way for me to send money to people I have sent money to previously and for making your user interface terrifying by not linking up names and phone numbers/Dwolla IDs.

Update: Dwolla contacted me soon after this post and put out an update to this issue the next week. Shout out to Dwolla!

Tor-ifying CasperJS

Over the past few months, one of my favorite tools has become CasperJS, which is a navigation and testing utility than runs on top of PhantomJS, a headless web browser. This is a great tool for doing web scraping, which you can use to automate the retrieval of data from webpages, among other things. Sometimes though, you want to test a target web page from a variety of different IP addresses, or find yourself behind a block of banned IP addresses, or just need to anonymize your activity. This is where Tor comes in handy.

Web scraping is a lot of fun, but make sure you are following the commonly accepted rules of web scraping:

  1. Make sure you’re following the target site’s Terms of Service. This means respecting robots.txt and any other restrictions there may be.
  2. Limit your requests. Scraping bots can navigate webpages much faster than normal humans, and you don’t want to accidentally DOS a site with an out of control scraper.
  3. Be nice to the server. If you don’t need images, modify your scraper so it doesn’t download images (PhantomJS has a --load-images=false flag for this). If you want to be really nice to the server, put your e-mail address in the scraper’s HTTP headers so the server admin can contact you if your scraper is giving them a problem.

(These rules were partially adapted from this list)

Now, on to how to “Tor-ify” CasperJS.

Download Tor

Well, obvs. I normally use REHL-based Linux distros, so I’m going to link to those instructions. Once torproject.repo is in your /etc/yum.d or /etc/yum.repos.d, you should be able to yum install tor with no problem. Then start the service with service tor start.

After you’ve confirmed that Tor can run on your machine, feel free to shut it down, as we’ll be coming back to that later.

Write a Script for Testing

How will you be able to tell that CasperJS is properly proxying through Tor? Why not write a script that scrapes whatismyip.com?

Taking a look at the source of whatismyip.com, it looks pretty simple to scrape. The IP address is contained in a div that has a handy id that we can pull data from. My workflow for writing scripts with CasperJS is to fire up the target webpage, write some code in Chrome Developer Tools, then copy that code back into my CasperJS script. Taking a look at the source for whatismyip.com, we can use some vanilla javascript to grab the IP address.

1
var ip = document.getElementById('greenip').textContent.trim()

As a force of habit, I normally add .trim() on to the end of any text I pull out of a DOM node since there’s no point in keeping useless whitespace.

Now, let’s star writing our CasperJS script.

1
2
3
4
5
var document=[]
var casper=require('casper').create({
  verbose: true,
  logLevel: "info"
});

I like to keep logging on for most of my Casper scripts, just because it’s helpful to see what’s going on and the output looks cool. Now we’ll define the first step of our scraping process:

1
2
casper.start("http://whatismyip.com/", function() {
});

This simply tells CasperJS to use PhantomJS to load up http://whatismyip.com.

An important note about CasperJS is even though everything is written in Javascript, you can’t actually manipulate or read the page you’ve loaded into CasperJS without running the evaluate function. So we’ll add to our first step:

1
2
3
4
5
casper.start("http://whatismyip.com/", function() {
      this.evaluate(function () {
          console.log(document.getElementById("greenip").textContent.trim());
      });
});

Once you’re inside this.evaluate, you have access to the document object. Also, notice how we do output with console.log, instead of this.echo. This is because once inside the evaluate function, you no longer have access to the this that refers to the Casper object.

Ok, so tying everything together for this really complicated script:

1
2
3
4
5
6
7
8
9
10
11
12
13
var document = [];
var casper = require('casper').create({
      verbose: true,
          logLevel: "info"
});

casper.start("http://whatismyip.com/", function() {
      this.evaluate(function () {
          console.log(document.getElementById("greenip").textContent.trim());
      });
});

casper.run();

You always have to put casper.run() at the end of your scripts to kick of the process of running through the steps. Now, if you’ll watch your terminal output, you should see your IP address output in the [info] [remote] section of the output. Yay!

Now, start up Tor again and we’ll pass in some parameters and see if the IP address changes.

Proxy PhantomJS

Tor is a SOCKS proxy, not an HTTP proxy. (Most of my previous exposure to Tor was through the Tor Browser Bundle, so this was interesting to me). But PhantomJS can also run through a SOCKS proxy, so no worries. Add these parameters when you start the script:

--proxy-address=127.0.0.1:9050 --proxy-type=socks5

(If you Tor proxy isn’t running on 127.0.0.1:9050, you should change that parameter. You’ll see where the Tor proxy is running when you run service tor start.)

So, running my final Tor-ified CasperJS setup looks a little like this:

casperjs --proxy=127.0.0.1:9050 --proxy-type=socks5 whatismyip.js

Note: while using Tor requests can take significantly longer to go through. I’ve seen some requests with this script take as long as 114 seconds to resolve, but such is the nature of Tor.

Won’t you do the world a favor and add a relay to the Tor network?

Creating Following and Followers Tabs With Flags in Drupal 7

I’m writing this post because I’ve been using Drupal to rapidly-prototype an MVP (and I have many, many thoughts on Drupal, but I won’t get into them here). One of the features was to have a list of followers (who follow a user) and a list of the users that a certain users follows, which is a very typical setup. I’m writing this post because I had one hell of a time putting this thing together in Drupal 7, and I’m hoping it will save someone else from going through the pain I went through.

First, let’s rubber duck and clearly define our requirements for the two lists.

Following: A list of users that a specific user has flagged ‘follow’. In other words, a list of users flagged by another user. Followers: A list of the users that have flagged a specific user to ‘follow’. In other words, a list of users that have flagged another user.

I’m using the terminology flag here since we’ll be using the flag module to put together the following/followers functionality. The two modules we’ll need are:

First, you’re going to need to create a new flag for ‘follow’. Setting up a new flag is fairly trivial, and I called my flag ‘follow_user’ and set it to be a global flag.

Now, create a new view for Followers. You’re going to need to set the following options:

And that’s how you do the followers tab. For the following tab, create another view, with basically the same parameters (the path and will be different, obvs), but the major differenct is with the Contextual Filter:

For reference sake, here is what the views look like in my admin:

Followers

Followers Settings

Following

Following Settings

Hope this helps someone!

An Overview of the Upcoming Facebook Privacy Changes

I can haz a vote?

Facebook is making changes to its Data Use Policy and Terms of Service (Statement of Rights and Responsibilities). Per the current policies, these changes have to be put up to a vote by Facebook users. Of course, the vote is only valid if 30% of the user base participates in the voting (that’s about 300 million people). At the time of this writing, only about 16 thousand users have voted and a majority of those (90%) have voted against the new policies.

In addition to minor changes clarifying data collection, privacy settings, and affiliates, the major change that comes with the new policies is the abolition of voting itself. If the new policies are passed, then there will be no more voting on new policies. Instead, there will be a seven day comment period before they are put in place.

Given the virality of the fake Facebook copyright notice, I’m surprised that the voter turnout is so low. On the other hand, Facebook has not done much in terms of publicising this effort. I suspect they want to do away with voting altogether, as it has never had a meaningful impact on the site’s policies and operations.

In lieu of a document comparing the two policies (other than this pdf from Facebook), I’ve created a little voters guide.

Here are the upcoming Facebook policy changes:

Data Use Policy

Signing up

Old Policy New Policy
Required to provide name, email, birthday, and gender. Have to provide information such as that in the old policy, but may use a telephone number.

Information received by Facebook

Old Policy New Policy
Does not mention Facebook’s affiliates. Language modified to include Facebook’s affiliates(see note)

Note on Affiliates Facebook defines affiliates to be business that are legally part of the same group Facebook is a part of. I take this to mean Facebook Ireland, which is a company set up to take advantage of Ireland’s tax laws, and Facebook Hypderabad.


Messaging (@facebook.com email address)

Old Policy New Policy
Contained details on how to control who messages you. Now anyone in a conversation can message you.

How Facebook uses your information New Policy: Insertion of a clause – “in addition to helping people see and find things that you do and share” – prefacing examples of how Facebook uses your information.


Timeline

New Policy: A reminder that even though you may hide a post, people may see it elsewhere, like on someone else’s timeline or search results.


Finding you on Facebook

Old Policy New Policy
Only friends will be able to find you via e-mail address or phone number, depending on your privacy settings People will be able to find you though a post to a public page or if you are tagged in a friend’s post or photo

Personalized Ads New Policy: Clarifies how personalized ads work. Sponsored Stores New Policy: Subscribers, in addition to friends, will see sponsored stories.


Data Retention New Policy: Facebook may retain information from suspended accounts for up to a year to detect repeat offenders.


Invitations

Old Policy New Policy
Facebook will send up to two reminders to friends you invite. Facebook will send a few reminders to friends you invite.

Affiliates New Policy: Facebook may share information about affiliates (see note above).


Opportunity to Comment and Vote New Policy: Voting is removed, as is the 7000 comment threshold for triggering a vote. Now users have seven days to comment before a change goes into effect.

Statement of Rights and Responsibilities

Your Facebook Timeline

Old Policy New Policy
You will not use a personal Facebook account primarily for commercial gain. You will not use a personal Facebook account primarily for commercial gain, and will create a Facebook Page to do so.

Promotions from Pages New Policy: If you run a promotion on your timeline from your page, you agree to the Pages Terms of Service.


Amendments to the Policy New Policy: Voting removed as well as the 7000 comment threshold (similar to the changes to the Data Use Policy). Seven day comment rule also applies here.

You can vote on these policies here.

I Haz Make Octopress Plugin

This week I’m deep in the process of converting a variety of PHP/MySQL backed sites (mostly Joomla and Symfony 1.4) over to Octopress, mostly because I don’t want to deal with the overhead of running MySQL (In the past, I’ve had to upgrade my Slicehost VPS in order to keep MySQL from hanging). Some of the pages had a little Facebook likebox included with them to collect likes. My search for an Octopress aside that would create a likebox was fruitless, so I made one.

How it works

Basically there is a plugin (likebox.rb) and an aside (likebox.html). To use the plugin/aside you move the files into their respective correct locations, then modify the default layout to load the likebox.rb plugin if Facebook likebox configuration is detected in _config.yml. The only reason there is a plugin is to load up the Facebook Javascript SDK code so the markup in likebox.html works correctly.

I’m wondering if there is a more elegant way to load up the Javascript SDK rather than just adding an if loop that will spit out static markup if true.

Check it out on github: octopress-facebook-likebox