Recorded Future Inc. is a software company based in Cambridge, Massachusetts, United States, specializing in web intelligence and predictive analytics. Using what they call a “temporal analytics engine”, Recorded Future provides forecasting and analysis tools to help analysts predict future events by scanning sources on the Internet, and extracting, measuring, and visualizing the information to show networks and patterns in the past, present, and future. The software analyzes sources and forms “invisible links” between documents to find links that tie them together and may possibly indicate the entities and events involved. The company was founded in 2008 by Christopher Ahlberg and has 16 employees as of July 2010. Both Google (on May 3, 2010) and the CIA have invested in the company, through their investment arms, Google Ventures and In-Q-Tel, respectively.
In a white paper, the company says its temporal analytics engine “goes beyond search” by “looking at the ‘invisible links’ between documents that talk about the same, or related, entities and events.”
The idea is to figure out for each incident who was involved, where it happened and when it might go down. Recorded Future then plots that chatter, showing online “momentum” for any given event.
“The cool thing is, you can actually predict the curve, in many cases,” says company CEO Christopher Ahlberg, a former Swedish Army Ranger with a PhD in computer science.
This seems to be a venture with a “product” for which there is no public market. It seems to be solely aimed at procuring data on individuals for spying use by the gov’t.
Also, it really reminded me of the “pre-crime” system in the film “Minority Report”. They relied so much on their “predictive accuracy” that they arrested people for a crime before it was even committed,and behaved as if the crime WAS committed.
The Swedish-American software company, Recorded Future, has developed a program that specialises in predictive analysis and with backers including the CIA investment arm In-Q-Tel, it is hoped that averting terrorist attacks could be one of its uses.
“It is a useful aid but prediction is always difficult. The world we face has wicked problems, and it’s important to understand that people like (Anders Behring) Breivik and Al-Qaeda are pretty savvy in concealing themselves through code words and disguised meanings,” says Magnus Ranstorp, Research Director of the Centre for Asymmetric Threat Studies at the Swedish National Defence College (Försvarshögskolan) and one of Sweden’s foremost experts on Islamic terrorism.
Recorded Future is designed by an ex-Swedish Army Ranger Christopher Ahlberg who also has a Masters in Computer Science.
While refusing to work with the Chinese government, Google has admitted its looking at this software which is just a teenie weenie step away from the infamous Al Gore Clipper Chip suggestion of the 90’s.
Using what they coin as a ”temporal analytics engine,” their forecasting tool helps analysts predict events and trends by scanning hundreds of thousands of high-quality news publications, blogs, public niche sources, trade magazines, government web sites, financial databases and more.
The program then sifts through the myriad of publicly available information, looking to identify links, extract information, measure data and visualize information that reveals patterns of the past, present and the future probability of almost anything.
The company claims its tool to be so sensitive to data changes that it could even ”beat the news” and anticipate political crises, disease outbreaks, major market fluctuations, resource shortages and natural disasters, to a name a few.
And the world is listening. Both Google and the United States intelligence agency, the CIA, are financial backers of Recorded Future through their investment arms by In-Q-Tel and Google Ventures, respectively.
The Massachusetts-based Recorded Future’s concept appears to be a close cousin to similar data analysis tools that Dow Jones, Thomson Reuters and Selerity are building for the financial industry. Their idea is to develop a program that analyzes the news to predict how it will affect individual stocks and markets.
While the company offers tailored approaches to the financial arena, it extends its ability to harness the predictive power of the web and can provide analysis for almost any sector or industry.
In this new era of redefined terrorism and the threat of seemingly random attacks around any corner, could such software alter history and evade or thwart evil plots?
“It’s important to have this technology, to use it, and to understand its limits. In the end, it’s not going to be enough though – we need human interaction to sift through data,” says Magnus Ranstorp.
Recorded Future claims it can monitor terrorist activity by tracking the individuals who are attracting attention, who are connected and where they are travelling, to cite a few examples.
Ranstorp, who CNN named its principal terrorism expert following the September 11th attacks and who was invited to testify before the first hearing of the 9/11 Commission, concurs that this type of network analysis can reveal strengths in patterns of different types of relationships between individuals and entities but he points out it is also always limited to open source data.
The Recorded Future team includes computer scientists, statisticians, linguists, technical business people with deep domain expertise in areas such as intelligence and quantitative finance. A high proportion of the team holds PhDs and other advanced degrees. Team members have received numerous accolades including the Fulbright scholarship, MIT’s TR100 award, and more. Our leadership has built multiple successful analytics businesses with aggregate annual revenues in the $100′s of millions. We have a great team, a solid track recorded of success, and a great culture.”
“Terrorist related operators know what not to say and use closed chat rooms. But, even when trolling open source information, one still needs to know exactly what they are looking for,” says Ranstorp, adding that due to the complicated nature of terrorism, one small change can change and effect many other facets in a blink of an eye.
“No intelligence service has the man-power or man-hours to go through the tsunami of information that is on the web, but it is still a useful tool to indicate broad-based trends and patterns,” he concludes.
Other sectors Recorded Future can assist relate to marketing and branding, public relations, business intelligence and more.
For example, it can monitor the impact of an advertising campaign or view the momentum of online discourse over a period of time for brand analysis.
Naturally, the program transcends language barriers with the ability to understand all the ingenious devices of the human language, amplifying its effectiveness.
Not only does Recorded Future review events and entities, it has the capacity to analyze time and space dimension, meaning references to where and when an event has taken place, or even where and when it will take place.
This is only the tip of its soothsaying abilities. The rest lies buried in the endless undulation of open source information swirling online.
Documents also have invisible links and the company believes it can track any document use as well as possible secondary issues such as relationships between disparate “things” – and people.
Example one : Bomber A to Bomber B to army support person X. The system will track this link and then analyse future threats.
Of course, it is not a perfect tool as some phenomena, like the spontaneous volcano eruption in Iceland, are truly impossible to predict.
While admitting their algorithms are not perfect, Recorded Future believes that by gathering, interpreting and linking the information available to the world, it can effectively reveal all that mankind really does know about the future.
Recorded Future charges a minimum of $149 per month for access to its data, and earns most of its revenue from selling to Wall Street quants and intelligence agencies. CEO and founder Christopher Ahlberg, a Swedish computer science Ph.D who previously created business intelligence company Spotfire, says that Recorded Future has no plans to launch a free consumer version of the service. Consequently, the company doesn’t do public relations and seldom talks to the press.
But last week Ahlberg sat down with SAI in San Francisco to discuss the company and — of course — his predictions for the future. Among the highlights:
When a company suddenly starts to get more press, its stock price tends to rise — even if the press is negative
In aggregate, stock prices go up 40 basis points in the week before earnings and down 40 basis points after (but to win at this kind of arbitrage game one would have to bet on every publicly traded company)
Recorded Future accurately predicted political trouble in Yemen about 15 months before it happened.
Here’s an edited transcript of our conversation:
Business Insider: Who are your customers?
Chris Ahlberg: We’re selling it to governments who are using this to do sort of intelligence analysis. Then we sell it to these quant investors, largely because we think of these as sort of low-hanging fruit, a good place to get started. This is a tough problem to solve. Long-term, we want to get it in the hands of comparative intelligence people. We’ve kind of done the anti-Silicon Valley strategy where most people, if they build something like we do, they put up a new search box and they say, “Go for it!”
BI: They give it away for free and they try to make money later.
CA: And hope for the best. It’s just very hard to do that. We sell access to our service at a minimum of $149 a month. We priced it at this salesforce.com pricing. Think of it as a competitive intelligence product. All kinds of pretty fancy software companies, high tech companies, automotive companies, and government agencies around the world have signed up for that. It’s pretty cool, actually. But it’s not this sort of “walk up and use it for free.” We might, over time, figure out some free offerings to rope people in.
BI: You don’t do a lot of marketing or advertising at all. How do you make that sales pitch? Coming at it from the outside, it sounds crazy, but based on past trends, we can do a pretty good job of predicting future events. How do you prove that?
CA: When you do this stuff in the stock market, there you have the truth. If you look to “I’m going to predict stock prices one week out,” or “I’m going to predict market volatility one month out,” our top level internal company objective this year was to beat the VIX in terms of predicting one month-out volatility. We’re actually doing pretty well on these things. We have a blog for that. We call it predictivesignals.com. You can go check it out. It’s very mathematical in its nature. It’s geared towards financial quants and all these guys.
It’s actually much harder to deal with these things that are infrequent. Can you predict how an election is going to go? Can you predict when an event is going to happen that has never happened before? A black swan event. My favorite example is when will Iran have nuclear weapons? Hopefully never. How do you define that? We have no clue. Do they already have them? Will it be 2014, 2015, 2016? We can’t make any pretense about that, but maybe we can help out with the analysis.
BI: Let’s take an Apple product announcement. What do you look at to get that data? [ed: This conversation was recorded the day before Apple unveiled the iPad 2 ]
CA: Let’s say that you work for a competitive mobile phone provider. You’re sitting there saying, “Dammit, these Apple guys win the marketing game. They’re shutting up completely, then they spread rumors, and the world doesn’t listen to anyone else.” Tomorrow the world will probably be more likely to listen to Apple to Libya news.
So what do people know? Tell me about any sort of product releases coming up in 2011. That will look for any hits in our world that will be where we found text that includes “Apple will launch a new iPad on Friday” or “in late November.” There’s been all these things that Apple might delay the product release until November or late September. So we timeline all that stuff for the user. They can sit down and take them apart, drill down into the underlying articles. And we try to score the events so that we give percentages to the stuff that has high credibility and the ones that don’t. So far I would call it manually assigned credibility, but we’re working on automatically assigning credibility on various topics. So if a source has been historically good that way, even if it’s not a simple one answer, we can help out this person who needs to dive in to this area to understand it to get a high-level sense of what the timeline of an Apple product release is like.
Then we can put them next to Android product release events. And give me a timeline of both. And then we can say, “Here’s a gap. Nobody else is launching in that gap. That’s your week.”
BI: The types of input you’re taking in – are these public media sources? Are these reports from financial analysts?
CA: Tens of thousands of sources going higher and higher. Eventually we’ll crawl the Web but for now we’re dealing with media sources. I would say on one end it ranges from Twitter to, on the other end, government filings and media and blogs and all kinds of things. Maybe from a credibility point of view you would put SEC at one end and Twitter on the other. We take, literally, tens of thousands of SEC filings, maybe more, per week. I’m not sure. Literally. The SEC is flooded with information.
BI: Have you read William Gibson’s last book by any chance? [Warning: spoiler coming.] He called the idea of predicting the immediate future “the flow.” And the industrialist, this guy called Bigend who’s been in the last three books, that’s what he’s going after. And he gets it at the end, basically predicting the future 18 minutes from now.
CA: What are his inputs?
BI: Exactly. It’s fiction!
CA: Our premise is sort of that the world knows a lot about the future. You have your own calendar in your phone, all around you. Newspapers contain simplistic calendarial information. There’s ton of this sort of microfuture. The future surrounds us. I gave a presentation and someone tweeted that. Kinda neat. What if we could organize that? It’s not gonna get to 100% in terms of outcome. I have no pretense of that. The human brain is pretty smart. Let’s pull that together.
BI: Do you find that Twitter is a useful source of information? When things are trending does that actually mean something?
CA: There’s a theory that says that stocks that get attention will be overbought and the price will rise. For example, Proctor and Gamble gets written about a lot in the news. A lot of people will flock to Proctor and Gamble and the price will rise. But when a price gets more attention in Twitter, there aren’t any compelling studies of that. You can use it as a way to measure amplification in certain sets of consumer-oriented companies and products – there’s a subset where it’s going to tell you things.
Likewise, search – Google has very elegantly proven that if you look up the number of people searching for an unemployment office, that predicts the unemployment number. Likewise searching for selling and buying a house. There are many other things where that doesn’t work, but there are things where it works very well.
BI: It seems like it works best when you’ve got very broad interest and a lot of inputs. But it would be very difficult to predict for a specific time or after a specific event. What’s the probability of a catastrophic hurricane flooding New Orleans this hurricane season?
CA: Even something like corporate bankruptcies. For those, people don’t spend time searching for that in Twitter. Things that are close to humans, where there is – it’s a different sort of thing. Unemployment, house, home, things that are close to you.
What’s exciting for us, we think, is that there are all these sorts of people writing about the future, saying, “This might happen.” Just aggregating that. Looking for attention. What companies are gaining attention, be it on Twitter or in SEC filings. Any company sticking out in SEC filings gains attention. Are there things that known experts start talking about? Or do known experts point to any sort of change? There are so many of these things where you can find glimpses of the future. And we’d like to organize all that and figure out ways to make it available. That’s what it all comes down to, and there’s a pretty damn nice opportunity there.
BI: Personally what do you think about the macro situation for the US for the next year?
CA: It’s tricky. The Fed has flooded the world with money, so that’s nice. I’m not going to sit here with the pretense that I can prove the money that the Fed has printed has moved into the $1.2 billion JP Morgan fund. Clearly there’s a lot of liquidity and part of it has moved into this part of the world. One way or another, when there’s lots of liquidity around, money has to chase something. It’s gone abroad a lot. It’s probably ended up here as well. You also have that money that’s getting 0% interest rates that’s gotta be put to use elsewhere. Then you counter that by the fact that you have more of a crazy outside situation – China and the US could end up in all sorts of situations.
We wrote great blog posts about about 14 or 15 months ago about Yemen being in trouble towards the second half of 2010. It took four or five months longer, but you go back and look back at our earliest blog posts, particularly the ones about Yemen, and basically about how people being poor and ending up in this sort of situation.
So you take those sort of things plus the money printing that could lead to unseen inflation. It feels sort of good now, but it’s the classic thing of feeling great now but not so great after a while. I’m probably very optimistic about 2011. I think it’s going to feel great and we’re going to have a lot of fun. But I’m less optimistic about are we going to end up in a debt situation that will be nasty, nasty, nasty? Probably two to three years out. Less optimistic, I must say.
BI: So, the question I ask everybody: tech bubble or not?
High tech, when money flows to it, it’s so much fun. I remember an old board member [at Spotfire] made the point that, “Dammit, I just knew that this was the party of the century.” Now I’m not going to get into any sort of analysis whether this is a bubble or not, that’s really hard. It has to do with money supply on some level. At some point, all this money flowing in has to be active. If you put money in something that is valued at whatever, and it doesn’t end doing well, people are gonna be a little pissed. I don’t think the people at pension funds who put money into private equity over the last ten years have been very happy. And I don’t know how many of them are going to be very happy coming out of this. It seems like the really good stuff is concentrated with a very small number of players. It can be very nasty on the other end of the spectrum.
BI: It seems like it’s a good time to be in it now.
CA: There’s probably also going to be one of these situations where it could very quickly go to “figure out a way to make money now.” I remember one of my best board members [at Spotfire] back in 2000-2001 told me “in 45 days we will be cash-flow positive.” That turned out to be incredibly good advice. And we walked away with $195 million in cash six years later.
BI: There was a time in between ’95 and 2000 where the smart money and the smart entrepreneurs were long gone and you had people who had no business running a company who somehow got funding to do it. I don’t think we’re anywhere close to that. The people are much smarter. Much more cautious. Everything’s staying private much longer. But eventually the dumb money comes in.
CA: Facebook makes money. Don’t forget that. They make more money than most S&P 500 companies. CEOs would be pretty damn envious of Facebook. Do we know their margins? It’s like 40-50%.
BI: $600 million operating income is what we’ve heard for last year.
CA: If that’s really true, then that’s mind-boggling. They’ve been cash-flow positive for a long time. Groupon was supposedly cash-flow positive after 7 months or whatever. I’m pretty envious of that too.