Open-Source Intelligence: public data for Intelligence Services (and also for Average Joe…)
Open Source Intelligence (OSINT) has emerged as a powerful tool in the digital age, transforming the landscape of information gathering and analysis. This article delves into the world of OSINT, exploring its applications, methodologies, and growing significance in various fields. From high-profile international events to cybersecurity challenges, we’ll examine how publicly available data is revolutionizing intelligence work. We’ll also discuss the ethical considerations and potential risks associated with these techniques. Join us as we uncover the intricate web of open-source information and its impact on modern intelligence practices.
Article originally prepared in Italian for my personal podcast Disruptive Talks (read it here).
Video version available here.
This content is also available as an audio podcast in episode S01E02 of Disruptive Talks on Spotify, Apple Podcasts, Deezer, Amazon Music, available here.
To begin this in-depth exploration, I pose a question: what is the common thread between:
- The death of Queen Elizabeth II
- The protests in Iran
- The Boston Marathon bombing in 2013
Have you found the connection?
Let me help you understand it. Let’s see together how technology allows us to link these stories.
On September 8, 2022, when news broke that Queen Elizabeth II was “under medical supervision,” some web users skilled in advanced research techniques immediately sprang into action. Using real-time private flight and jet trackers, they noticed unusual air traffic around Scotland, where the Queen was at Balmoral Castle. In particular, Prince William’s plane was tracked. These movements revealed that the situation was more serious than anticipated. Indeed, a few hours later, the Queen’s death was announced. A death that some had already anticipated…
During the 2022 protests, Iranian activists have identified some agents involved in the repression by cross-referencing videos and images with social media profiles and other open sources. For example, an agent who beat a protester was identified by comparing his face with his WhatsApp profile picture. In another case, they managed to confirm the killing of a 16-year-old protester, Nika Shakarami. By analyzing videos and photos, activists reconstructed her movements before her death, refuting the Iranian authorities’ version that she had run away from home.
On April 15, 2013, two bombs exploded near the finish line of the Boston Marathon, killing 3 people and injuring 264. In the days that followed, investigators were groping in the dark. But on Reddit and sites like 4chan, ordinary users spontaneously set to work analyzing photos and videos of the attack spread on social media. In particular, efforts focused on images showing two men with identical backpacks, captured before and after the explosions. Users enlarged the images, compared faces pixel by pixel, and finally, after some failed attempts, identified the suspects: brothers Dzhokhar and Tamerlan Tsarnaev. All this while the FBI was still in the dark. Only on April 18 did the FBI release photos of the attackers, whom online users had already discovered the day before.
In short, in all these cases, crowdsourcing and open data analysis proved faster and more effective than professional intelligence. These techniques and methods, which allow anyone to conduct open and transparent investigations into global events, fall within a field called OPEN SOURCE INTELLIGENCE, often abbreviated as OSINT.
So, OSINT is the art of collecting and analyzing publicly accessible information from open sources such as websites, social media, public data, satellite images, and much more. This information can reveal useful patterns and connections for a variety of purposes, from investigations to cybersecurity.
But what exactly is meant by “publicly accessible data”?
Let’s start with the simplest and most immediate examples.
Search engines like Google, Bing, and Yahoo are obviously a goldmine for OSINT. Just do a search with a person’s name to find websites, social profiles, articles about them.
Social networks themselves are also a valuable source. Users often share personal information, such as the city they live in, where they work, even when they’re on vacation. Just take a look at a target’s profile to discover many things.
But OSINT goes well beyond traditional search engines. For instance, specialized search methods, often referred to as “Google dorking“, leverage advanced search operators to uncover sensitive information such as PDF documents, presentations, and spreadsheets. Tools like Google Dorks can help you find this information, potentially revealing leaked resumes or financial data online.
The deep web and public datasets are also rich sources for OSINT. Tools like Maltego or SpiderFoot help in mapping information like land registers, court archives, and financial documents—data that is theoretically public and legally obtainable, even if buried behind complex search interfaces.
When it comes to technical sources, metadata analysis tools such as ExifTool can extract information from digital photos, revealing the GPS coordinates, time, and even the smartphone model used to take the photo. This kind of analysis is invaluable in verifying the authenticity and context of images.
Moreover, analyzing a website’s code can disclose the technologies deployed and highlight potential vulnerabilities. Tools like Wappalyzer can identify technologies used on websites, while security-focused applications like OWASP ZAP provide insight into possible vulnerabilities by simulating attacks.
In summary, the variety of OSINT sources is virtually infinite. As are its fields of application, which cover practically every sector with information gathering needs.
Obviously, intelligence and national security, but also law enforcement and private investigators. Cybersecurity experts use OSINT techniques to analyze online threats, anticipate cyber attacks, track hackers, and prevent sensitive data breaches. Companies, to obtain information on markets, competitors, potential partners. Investigative journalism leverages OSINT to cover war events, scandals, and cases of public interest.
In this regard, I recommend the excellent Bellingcat site, long a forerunner in the practice and training of Open Source Intelligence.
But why talk about OSINT today, in 2023, 10 years after the Boston bombing that popularized these techniques?
OPEN SOURCE INTELLIGENCE is becoming increasingly central in the era of big data and online information overabundance. Every day we produce 2.5 quintillion bytes of data, an unimaginable amount. Much of this is public and accessible.
But it’s not just the large amount of data that allows the development of this discipline.
The use of drones and other aerial surveillance devices is increasingly widespread among OSINT investigators. Drones equipped with high-definition cameras, such as those from DJI, enable close-up views of otherwise inaccessible areas, providing invaluable images and videos for investigations. This capability has been utilized in situations like monitoring illegal deforestation or gathering intelligence in remote conflict zones.
Artificial intelligence plays a pivotal role in processing and analyzing vast volumes of data in real-time, identifying patterns and insights beyond human capabilities. Tools such as IBM Watson can connect diverse data sources and extract intelligence automatically. For instance, AI-driven analysis can help identify trends in social media or predict stock market fluctuations based on news headlines.
Blockchain technology, known for its transparency and traceability, is an area of interest for OSINT analysts. Platforms like Elliptic are used to trace blockchain transactions, which is crucial for tracking hackers or identifying individuals behind public keys. This has proven effective in cases involving cryptocurrency theft or fraud.
Deepfakes, or videos manipulated using AI, represent the cutting edge of OSINT efforts in combating fake news and misinformation. Technologies like Deepware Scanner are being developed to detect such manipulations. Unmasking deepfakes has become crucial in political and diplomatic spheres to maintain information integrity.
In cybersecurity, OSINT is becoming indispensable. Tools such as Shodan can collect vast amounts of public information for analyzing cyber threats, identifying security vulnerabilities, tracking hacker activities, and preventing potential attacks. For example, OSINT techniques are used to sift the dark web for sales of zero-day exploits or stolen credentials. Monitoring hacker forums and communities with platforms like Recorded Future can reveal early signs of phishing or ransomware campaigns.
Additionally, by analyzing open-source code repositories on GitHub, researchers can uncover vulnerabilities in widespread software libraries, helping companies safeguard their systems. For instance, the discovery of the Heartbleed vulnerability in OpenSSL was facilitated through such initiatives.
Automated OSINT tools like GreyNoise monitor domains, IP addresses, and web pages for malware or anomalous behaviors, alerting organizations to potential threats. These proactive measures are vital for enhancing cybersecurity defenses.
Finally, social engineering techniques, such as simulated phishing attacks via platforms like PhishMe, test user awareness and the effectiveness of security controls. By simulating real-world cyber attacks, organizations can improve their preventive measures and user education programs.
Personally, I’ve found OSINT very interesting for several years. Reading Bellingcat articles, where journalists detail investigative techniques, is always stimulating. If the topic intrigues you, I also recommend the OSINT Framework, a collection of OSINT sources and techniques classified by categories.
For everyone to try, I recommend looking into Google Alerts for setting up notifications about specific topics or people, Have I Been Pwned to check if your email or phone number has been compromised in a data breach, and ExifTool for viewing the metadata of images for clues like location and device details. TinEye is great for reverse image searches to detect altered images or find image sources, and tools like Hootsuite can be used for social media monitoring, helping individuals keep track of mentions and relevant trends. Additionally, theHarvester is excellent for gathering emails, subdomains, and open ports, while Recon-ng provides a powerful framework for open-source web-based reconnaissance. These tools are user-friendly and provide valuable insights without requiring advanced expertise.
In conclusion, OSINT is a constantly evolving discipline, with new tools and techniques emerging. For journalists, investigators, analysts, or even simple curious minds, it offers unique opportunities to shed light on obscure events and bring hidden truths to the surface. But let’s remember to always act ethically and responsibly. After the Boston bombing, some OSINT analysts publicly identified a completely innocent person.
So, as Stan Lee would say: with great power comes great responsibility.
Feel free to share questions and advices in the comment section. For particular demand / request, reach me out by email or through the contact form on this page.
Other links
- Rand
- https://techjournalism.medium.com/how-irans-internet-block-gagged-local-online-protests-5dcf5dfa0d19
- https://www.csis.org/analysis/protest-social-media-and-censorship-iran
- https://www.theatlantic.com/national/archive/2013/04/reddit-find-boston-bombers-founder-interview/315987/
Tools