Dave's Adventures in Business Intelligence

SAP had some fun on the BI 4.0 launch in New York last February. For years SAP along with other vendors have been touting their ability to import and analyze external data from various social media sources. Two SAP presenters at the launch event took a vote via Twitter as to which tie would meet the “Scissors of Destiny” at the end of the session. (Steve Lucas made an impassioned plea to save his tie, which he said was a gift from his wife, versus Dave’s tie which he “… just bought last night.” Steve won, and his blue tie survived.) It was a fun display of technology, but is it really that important? How impressive would it have been if the “fail whale” had picked that moment to make an appearance?

I don’t usually spend a lot of time here on my blog talking about philosophical aspects of BI as I am personally more interested in technical issues and solving problems. But the apparent consensus as to the importance of social media bugs me.

Let’s Start With Twitter

For example, I’m not a big fan of Twitter. I only follow a few folks, and I don’t tweet much. Yet despite my lack of interest Twitter was in the news the other day for setting a record number of tweets during the final for the women’s world cup soccer tournament. The record (which is sure to be broken) was over 7,000 tweets per second during the tournament. 😯 That exceeded the rate during the last Super Bowl (United States professional football championship in case anyone doesn’t know…), the recent royal wedding, even the announcement of the death of Osama bin Laden.

The exciting climax drew 7,196 tweets per second, according to Twitter. Paraguay’s penalty shootout win over Brazil in a Copa America quarterfinal later the same day came close to beating it with 7,166.

The previous record of 6,939 was set just after midnight in Japan on New Year’s Day. Other spikes include bin Laden’s death (5,106 per second) and the Super Bowl in February (4,064).

Read the full article at ESPN.com for more details.

Seven thousand tweets per second. As a data point showing the growth of Twitter that’s certainly impressive. But how many of those tweets contained information that was valuable and relevant to what I really wanted or needed to know? How am I supposed to filter or otherwise process all of that information? Or put another way, how much of that data was signal, and how much was noise? During events like Tech Ed I already don’t tweet much because I would very likely be repeating something everyone else is already saying. Try this: search for the SAPTechEd hashtag, and see how many tweets that you find that are essentially the same. I think most folks would agree that quantity is not the same as quality, yet as a society we seem obsessed with bigger / better / faster so numbers impress us. Seven. Thousand. Tweets. Per. Second. Wow. 🙄

Some may argue that it’s not the individual tweets that become valuable at that point, it’s the overall sentiment displayed in the tweets that becomes interesting. That’s a valid point, and that’s something that the Text Analytics product (now integrated into the Data Services suite) was designed to do. High-level analysis of high-volumes of data makes sense, but I certainly can’t use it on a personal level.

Along those same lines (and related to the “my followers list is bigger than yours” competition) did you know that you can reportedly buy Twitter followers now? (Seriously, google for “buy twitter followers” and see what you find.) Why? Because people with “presence” (and to some companies presence can be defined by the size of your followers) can get paid for tweets. The Kardashian sisters are apparently masters of this, rumored to get paid in the neighborhood of $10,000 (some sites suggest even more) for a single tweet. I wonder how many followers I would have to buy before getting to that territory? What would the return on that investment look like? I remember getting excited – briefly – when I hit 100 followers. Then I went out and looked, and about 20% of them were spammers, including a number of “sweet young things” that wanted me to visit their web sites. Why would someone like that follow me? Especially when Jamie Oswald is obviously much more attractive? (He must be, he has three times as many followers as I do, not to mention 60 times as many tweets. Wait, what was that quality versus quantity thing again? :P)

In Which I Personally Declare 99% Of The Internet To Be Crap

The Internet is a wild place where what few rules might exist are not always followed. If there is money to be made, then someone will figure out a way to abuse the system. It’s not just the “little guys” either, as evidenced by the way retailer JC Penney took specific steps to outwit Google during the 2010 holiday shopping season. Google unwittingly initiated this years ago when they decided that their search engine would give bonus points to web sites that had more incoming links (also known as “back links” because they link back to a site) than other sites. Entire industries came into existence (SEO or Search Engine Optimization for one, link farming for another) that didn’t exist before, simply because there was money to be made. So why do spammers “follow” people on twitter? Because it’s yet another way to establish a link back to their own web site. Even if Google discounts the link, some of my other followers might be curious enough to click a link, and often that’s enough for the person on the other end to make a few pennies. Make a few pennies often enough and it can add up to real money after a while.

On BOB we are fighting this same battle and have been for years. In the early days of our community we were getting dozens and then hundreds of new members every week that signed up but never activated their accounts. Or perhaps they activated their account but never logged in to post anything. Why? It’s for the same reason, they wanted back-links to their spam web sites. Why would anybody waste time to do this? The reality is that they don’t. No single person would sit down one morning and decide to go out and register on hundreds of different discussion boards, or post comments on hundreds of different blogs. What happens instead is that a single person will instead sit down and decide to write a script that scans the Internet looking for discussion boards or blogs. Once they find them, then perhaps another script is used to go out and register and/or post on that board or blog. Last month (September, 2011) the BOB software blocked or rejected almost 20,000 such attempts. That’s over 650 attempts per day that were rejected. Numbers were even higher in previous months such as June when we blocked 23,514 similar attempts. In May it was 23,729 attempts.

Blogs are not immune. This blog ignored or denied an average of over 6,000 comment attempts per month over the last six months that were suspected to be from spammers. It’s my script fighting against their script, and it’s the “wild, wild west” all over again. As I said earlier this is not a new problem. Back in 2008 I wrote a blog post showing that spammers could have potentially cost me in the neighborhood of $20,000 had I not come up with my own automated tools to combat them.

Even Big Guns Like Google Are Not Immune

Google’s gmail service is overrun. In the past year, Google’s gmail has been used more often by spammers (based on my own spammer logs) than the next five service providers combined! That includes some service providers that folks probably know (yahoo.com, aol.com), some that board and blog owners around the Internet have come to know and despise (mail.ru, gawab.com), and some that are just plain humorous (hamstermail.net and anotherspamdomain.org). No, I did not make that last one up. Gmail is not a recent problem either. Here are some numbers that I compiled back in 2009 for another site. Want to read more about web-bots and how they post while you’re sleeping at night? Here are links to “Are You A Zombie?” Part I and Part II, written – appropriately enough – on Halloween of 2008. This is not a new problem, and it continues to get worse.

It turns out that computers may be even better at solving visual CAPTCHA puzzles than humans are! A CAPTCHA or “Completely Automated Public Turing test to tell Computers and Humans Apart” is that annoying series of distorted letters and/or numbers that I sometimes have to decipher in order to complete a web form.

Trust No One

Given all of this background, almost nothing blows my mind more than seeing a web address at the end of a TV commercial that links not to the advertising company’s home page but to Facebook! Why? Why are companies (and big companies too, not small ones) spending millions of dollars to build someone else’s brand? (Facebook in this case.) Beyond that, why would I trust any external entity to collect and manage what could very well be critical data from my (current or potential) consumers? And how do I know that the data you’re collecting is even valid?

There are rumors that Sarah Palin got caught setting up a secondary Facebook account, just so she could “like” herself and skew the results shown on her main page.

Why Worry About All Of This?

Aside from the stuff I have already mentioned about having to deal with spam blog comments, spam twitter followers, or spamming BOB members, is there anything else to be concerned about? Wasted time is wasted time, sure, but IT folks have become quite adept at identifying and eliminating email spam, and I’m sure we’ll get better tools to manage blogs and other social web sites as well. But I don’t think the problem stops there.

What if I am the target of an organized but subtle campaign?

Suppose I have a company… one that might be familiar: the ACME Widget Company. We make the best widgets the universe has ever seen. However, I have a major competitor who is trying to impact and they’re not shy about which methods to use. They found out that I was running a public opinion poll as to what popular color I should include in my next widget catalog. They hired some less-than-ethical programmers with plausible deniability and charged them with skewing the results of my public polling. With a little strategy, these hackers are able to carefully do so. Votes came from all over the world (via the already established botnets) and at all hours of the day. They carefully spread out the votes so that statistical analysis will not reveal any odd patterns. As a result, my new widget color for 2011 is hot pink. I produced tens of thousands of hot pink widgets, only to find out that nobody really wanted them. Is this a far-fetched scenario? Perhaps, but maybe not.

Granted an online survey should not be the only way I use to determine a new product color… but what if it was? Could we have ended up with pink M&M’s instead of purple? Would Steve’s tie still have survived if Dave been able to whip up a Twitter-spamming script to skew the votes?

What’s My Point?

Is there a point to all of this ranting about spammers and botnets and hactivists? In my opinion, despite SAP and others appearing to really want to make social media relevant, I find myself deciding that I’m not ready to trust it, not just yet. Hot topic for the day, certainly. Cute marketing gimmick with the ties? Sure. Critical success factor for my business? Not so much. With the anonymity of the Internet anyone can post anything anywhere they want, and with the help of some friendly spammers, as much as they want. It almost seems to me that people have forgotten how bad the email spam problem was years ago because IT folks and ISP’s have become fairly adept at identifying it and filtering it out. Social media services like Twitter and Foursquare and others are all going to have to figure out the same thing or they will be overrun just like Google’s gmail service already is.

As an aside: I talked to Timo Elliott at a conference a few years back and I asked him how he dealt with what surely must be hundreds if not thousands of “link requests” on LinkedIn from folks that he didn’t know. He shrugged and said that he just ignored them. This despite the fact that LinkedIn is a relatively closed system, unlike Twitter which is wide open for everyone to see and (ab)use.

If I seem bitter or disillusioned it’s probably because I know what goes on behind the scenes on my blogs and on the boards that I manage. As a result I have become more skeptical and much less trusting of data sourced from the Internet. Don’t even get me started on privacy issues related to social media sites (Facebook, Twitter, Foursquare, even the new Google+) as that’s an entirely different rant.

That’s my opinion today. What about five years from now?

Five years in Internet time is a lifetime. I can’t guarantee what it will be like then, but I can guarantee it will be different that probably anybody expects. Will Facebook still be here, or will it pull a MySpace and disappear? Will Twitter implode under the weight of millions of tweets, or more importantly will it finally find a way to make money? More importantly, will anybody care if they do?