Palantir: The Seeing Stone of Security and Defense Analytics
Sifting through government data is no easy task.
Note: for the purpose of this piece, I’m focusing on Palantir Gotham, the firm’s government-facing arm.
In J.R.R. Tolkien’s Lord of the Rings, a palantir is a magic crystal ball that enables its owner to communicate over great distances and perceive current or past events. In Tolkien’s universe, the palantiri made what would ordinarily take weeks or months instantaneous and conferred great advantages onto those who held one. In that regard, Palantir Technologies is aptly named.
Palantir sells software that digs through massive data sets to generate meaningful, actionable information served up in intuitive ways. In addition to providing services for private sector clients like hedge funds, Palantir services a veritable alphabet soup of government agencies – CIA, DHS, NSA, FBI, and DOD, to name a few (in fact, an early investor was the CIA’s venture arm, In-Q-Tel). This is no easy task. Selling to defense and security agencies is an extraordinarily difficult, often relationship-driven process that is dominated by players like Lockheed Martin and Raytheon.
https://www.youtube.com/watch?v=9ooAUeTlzdU
Dealing with the raw data is another monumental challenge. First, there is too much of it. A single UAV can collect several terabytes of data every minute. Second, the data sets are messy, a product of inconsistent reporting, disparate reporting systems, and constantly changing technologies. Finally, the data are organized by clearance level, ranging from unclassified to top secret, which further obfuscates things. Despite these challenges, Palantir is thriving, having secured $1.2B in federal contracts since 2009.
The firm’s products fill a critical need for defense and security customers. Consider the following anecdote. During the conflicts in Afghanistan and Iraq, service members collected volumes of information from detonated and undetonated improvised explosive devices (IEDs), sensitive site exploitation, surveillance activities, and biometric analyses. For example, if an IED were found, an exploitation team could pull fingerprints and record serial numbers of electronics or munitions with the hope of tracking down the responsible party. Unfortunately, that data was not stored centrally, and any attempt to produce actionable intelligence had to be done manually. Palantir partnered with the US Marine Corps in 2011 to develop an advanced analytic capability that enabled intelligence analysts to access and manipulate data in an intuitive, easy-to-visualize manner. The results were stunning: analysts mapped out bomb-making networks, linked devices to individuals, and detected correlations between variables like weather and IED-type. Watch the video below for a demonstration of the software (note that it’s from 2011).
https://www.youtube.com/watch?v=lLRTSXjSHNY
Palantir continues to face challenges as it looks to generate new business with the government. As a Silicon Valley startup, competing with entrenched defense firms remains an obstacle, since government procurement processes tend to favor incumbents. Palantir recently sued the US Army for fair consideration in bidding for a $200M contract related to an intelligence-gathering system. The firm may have been dismissed by the Army in part because of a cultural divide: Palo Alto executives rocking slacks and semi-unbuttoned dress shirts didn’t make a great impression in a 2009 meeting with generals and high-level government bureaucrats. Moreover, Palantir has had a tough time hanging onto employees impatient for an IPO, and in the wake of Edward Snowden’s NSA revelations, it has taken considerable flack over its involvement with the agency’s spy program. Regardless, the company has compellingly demonstrated the way in which data and analytics can create huge value for government and defense.
Really interesting post James. The government is a treasure trove for any data enthusiast! I’m surprised to see how well Palantir is actually doing with the government. They are bringing real innovation to government intelligence through big data. My major concern here is with their ability to work with other governments though. Are they restricted to working with the US government? Companies such as Palantir that have large data sets usually learn things that they are then able to transfer to other clients. I imagine if they worked with China, the US government wouldn’t be so happy about it and would probably be nervous about data breaches. I think the government still underestimates how valuable big data is and they might find out how big of a threat it is if it gets into the wrong hands.
Great post, James. Palantir has truly redefined how the government uses structured and unstructured data. Do you think there’s a world where a competitor could pop up and potentially steal a government contract? Or is Palantir’s reach already too entrenched in these departments? Also, this question is outside the government arm but curious to get your take on how they’ve expanded into other industries in the private sector and how sustainable this will be long-term.
I really enjoyed this post James. I’m very curious though about Palantir’s ability to scale. They are really more of a consulting agency than a software company, because the types of contracts they pursue are so disparate is seems hards to have any overlap where they can do the next project for cheaper. They seem to have a huge edge in recruiting top people from math and statistics fields, but ultimately every project they will pursue will be hugely labor intensive.
Great post, James. Palantir certainly seems to have capitalized on an “institutional void” in the government so far. Your post seems to suggest that it might be time for them to diversify beyond government related work. Given geographic expansion could be challenging, I wonder if focus more on US financial markets might be more be fruitful (though also much more competitive).
Palantir’s use cases are certainly impressive, but it’s taken me some time to better understand what the company does. A visit to their website is not very helpful – just a “meta” plug for helping companies transform the way they use data. Interestingly, they do not really appear to be a big data company the way the press often labels them. Their core focus seems upstream from advanced analytics, closer to the plumbing. They help companies connect all their disparate data bases and make them readily accessible for employee’s queries. A key insight of theirs may be that most companies need to clean up their data before they can really dive into it.
Great post, James. It’s interesting that the presence of Palantir’s software in the hedge fund space is never really mentioned in the public buzz. I’m glad you brought that up! Given Prof. Viceira’s argument that the “cumulative alpha” in the world is reaching a plateau, do you think Palantir is the best positioned to address it? or someone else?
Super interesting stuff. Palantir has had some public shakeups with clients who have decided not to continue past the pilot phase of an engagement. Wonder if this means Palantir is best applied for some verticals and not others. Would be interesting to know what ROI Palantir is able to demonstrate for its successful client engagements vs.the others.
Very interesting post James, thanks for sharing! While working with the government gives them credibility I am worried about their ability to scale, especially internationally. Also, do you think government involvement would constrain their ability to innovate or undermine their independence in decision-making?
James, thanks for the great post. Palantir is certainly an interesting company. I am curious what your thoughts are on the role that Palantir has played in helping expand and accelerate the NSA’s global spy network, and the broader implications for “technology” firms such as this could be playing in allowing global governments to intrude upon the privacy of the very citizens that they are purporting to protect. Palantir has frequently stated its commitment to protecting the “fundamental rights to privacy and civil liberties”, and yet, its partnership with the NSA has included building tools to enhance the use of XKEYSCORE, one of the most intrusive tools wielded by the NSA, capturing nearly everything that users typically do on the internet by collecting emails, chats, pictures, web searches, social media traffic, etc. At a time when many tech companies are pushing back against global governments to defend users’ privacy (e.g. Apple), it seems that Palantir’s products have been specifically designed to co-opt any hope of private citizens having true digital privacy.
Thanks for your post, James. Indeed Palantir seems to be one of the most powerful technological tools that have emerged over the last 20 years or so. When I think about their business, I think about separating it into two parts: the amazing data synthesis and merging of disparate data sets into one, and the second being the analysis and insights into that data. I can definitely see how the first part would scale fairly well, but I am curious and wonder how challenging it is to scale the second part – analysis and insight. I would argue this is where Palantir can capture the most value comparatively, but how much do you think Palantir should balance creating custom high-value products for its customers? Or building a use-by-all platform that scales very easily?