{"id":138,"date":"2021-11-13T14:52:44","date_gmt":"2021-11-13T14:52:44","guid":{"rendered":"https:\/\/usersearch.org\/blog\/?p=138"},"modified":"2021-11-15T15:20:41","modified_gmt":"2021-11-15T15:20:41","slug":"how-to-build-a-web-crawler-for-osint","status":"publish","type":"post","link":"https:\/\/usersearch.org\/blog\/index.php\/2021\/11\/13\/how-to-build-a-web-crawler-for-osint\/","title":{"rendered":"How to build a Web-Crawler for OSINT"},"content":{"rendered":"\n

How exactly can you build web crawlers to help with OSINT Investigations? What an amazing question. This is a question we asked ourselves over a decade ago. Hence…the birth of UserSearch.org!<\/p>\n\n\n\n

In this article, we’re going to talk a little about how you can build your own web crawler using Python (don’t worry, it’s going to be basic). After 10 minutes of reading, you may even have your own Python web crawler to call your own!<\/p>\n\n\n\n

We’ve been asked a number of times how our reverse username search works<\/a>.  Trying to explain it causes a blank face.  Instead, were going to do a post on how web crawlers can be used for open-source research (ok and how our website works).<\/span><\/a><\/p>\n\n\n\n

Health Warning<\/em>: Very technical.  Requires knowledge in PHP and Python.  If you don’t have it, keep reading and you may get an idea of how search engines work (or ours at least).  Strongly recommend you read our previous posts first (Does a website know you?<\/a>, Can you find my hidden email address?<\/a> and New Search engine on the blog<\/a>).  It will give you a grounding on how the manual techniques work, so when you start reading our code – it will click!
<\/em><\/p>\n\n\n\n

What is a Web-Crawler?<\/strong><\/h2>\n\n\n\n

First, what is a web crawler? (as per Wikipedia https:\/\/en.wikipedia.org\/wiki\/Web_crawler).<\/p>\n\n\n\n

“A Web crawler is an Internet bot<\/a> which systematically browses the World Wide Web<\/a>, typically for the purpose of Web indexing<\/a>. A Web crawler may also be called a Web spider,[1]<\/a> an ant, an automatic indexer,[2]<\/a>, or (in the FOAF<\/a> software context) a Web scutter.[3]<\/a>“<\/p>\n\n\n\n

Types of Crawlers?<\/strong><\/h2>\n\n\n\n

This explanation is by no way official but from how our site works, we believe there are three distinct crawler bots.  There are crawlers that are extremely sophisticated and crawl, indexes, and remember where it’s been (google for example), and there’s directional.  Directional crawlers are more specific and have a particular task that needs completing.  Still very sophisticated, but designed for a handful of specific, targeted results.  <\/p>\n\n\n\n

Usersearch.org has been designed mostly on directed crawlers.  We’ve built the crawlers from the ground up, a blank page.  As you can see, our web crawlers have been specifically designed to find user names, pseudo names, email addresses, phone numbers, and website social stats across approximately 500 social networks and forums.  And finally, there are Omni-directional crawlers that are extremely specific with no wiggle room, a bit like accessing an API with very few moving parts.  We use a few of these too.<\/p>\n\n\n\n

Techniques<\/strong><\/h2>\n\n\n\n
\"Reverse<\/a>
Reverse lookups are the cornerstone to search engines. They all started from web crawling technologies.<\/figcaption><\/figure><\/div>\n\n\n\n

If you’ve read our previous posts (Does a website know you?<\/a>, Can you find my hidden email address?<\/a> and New Search engine on the blog<\/a><\/em>) you will see the manual process is really not that difficult to work out if a website knows a particular email address.  But, if you start doing that process manually on 10, 15, 20, or 100+ websites – it gets boring, fast.  <\/p>\n\n\n\n

The solution to this, of course, is to build a directional-web crawler (our own definition!).  Directional-Web crawlers (a bit like a Directional satellite where a line of sight must exist for two satellites to communicate).  Our directional web crawlers know exactly where to go, where to look, what to do, and where to put it.  Of course, we need to put some safety measures in place for various conditions such as an unexpected change in the targeted web page or a page that is temporary not responding.  But that’s just part of the fun.  We need to let the crawler know what to do, should some data get clogged into the system that it may not have expected (such as a space between two words such as Fred Hammer, rather than Fred_Hammer).<\/p>\n\n\n\n

Basic Python Modules for web-crawling<\/strong><\/h2>\n\n\n\n

So, we are not going to cover how to install python or how to test the modules.  We’re hoping you already know this (if you don’t, we can do a future post if you ask us).  We’re jumping right in.<\/p>\n\n\n\n

Good web crawlers technologies:<\/p>\n\n\n\n

\"Anyone<\/a>
Anyone can build an OSINT lookup tool with a little coding<\/figcaption><\/figure><\/div>\n\n\n\n

-Scrapey<\/p>\n\n\n\n

-Silenium<\/p>\n\n\n\n

-Mechanise<\/p>\n\n\n\n

Scrapey we don’t like too much as it tries to do everything for you.  We’d class this as a generic web crawler (not directional \/ omnidirectional).  So it’s not much use for us, but it’s good for mass-web crawling projects.  <\/p>\n\n\n\n

Silenium is great if you want a ‘point and click interface (Omni-directional).  You can build a little program in a matter of minutes that will do simple actions like entering you’re credentialed into a web-based email account, signing in, and sending email repeatability.  Pretty cool as you actually see the actions taking place (mouse movement, firefox opening, page loading, email being typed, etc).  Good for presentations.  <\/p>\n\n\n\n

Mechanize (Omni and directional) is a module that allows you to interact with websites similar to Silenium but in the background.  This means you can multi-process thousands of iterations at the same time, independently of each other (we use mechanise).  You can then take the data captured from the crawler and use the power of python to interact with the data or store it in a database.  It’s probably the best free solution on the market if you can use python.<\/p>\n\n\n\n

Python Mechanise Basics<\/strong> – crawling the web<\/h2>\n\n\n\n

Mechanise can browse to a web page and access a specific web form, and then enter details into that form and submit.  It can then take the result of that form submission and do something else with it – whether that stores the result in a database or keep filling in the next page of a form and continue.<\/p>\n\n\n\n

The below python code simply starts to mechanise by creating a mechanize object (br = mechanize.Browser() ) and then opens a website (response = br.open('some_site');.<\/code>  The program then goes on to list all links on that page (br.links ).<\/p>\n\n\n\n

From here it starts a ‘for’ loop that just iterates through each link found on that page, opens that link, and lists all the links of that opened page.  There you have it, in under 10 lines you have a web-crawler that will open a webpage and crawl all the links on that page and then continue onto the next links and continue.  It’s basically walking through every single page on a particular page.<\/p>\n\n\n\n

**************************************************************************************************************<\/p>\n\n\n\n

br <\/span>=<\/span> mechanize<\/span>.<\/span>Browser<\/span>() # Creates object<\/span>\nresponse <\/span>=<\/span> br<\/span>.<\/span>open<\/span>(<\/span>'some_site'<\/span>); # opens site and puts the value in 'response' varable<\/span>\n\ncurrent_links <\/span>=<\/span> list<\/span>(<\/span>br<\/span>.<\/span>links<\/span>()) # list links<\/span>\n\nfor<\/span> link <\/span>in<\/span> current_links<\/span>:<\/span>\n  br<\/span>.<\/span>follow_link<\/span>(<\/span>link<\/span>) # opens link<\/span>\n  sub_links <\/span>=<\/span> list<\/span>(<\/span>br<\/span>.<\/span>links<\/span>()) #get links from opened page<\/span>\n  for<\/span> link <\/span>in<\/span> sub_links<\/span>: # opens the next lot of links<\/span>\n    br<\/span>.<\/span>follow_link<\/span>(<\/span>link<\/span>) # follow the next lot of links<\/span><\/code><\/pre>\n\n\n\n
**************************************************************************************************************<\/p>\n\n\n\n
Cheatsheet for building quick web crawler functions<\/strong><\/h2>\n\n\n\n
So the above may be a little confusing.  So below is a step-by-step guide on creating a crawler that will enter some details into a form and then submit it.  You can see from the below that you need some HTML knowledge in locating the forum variable names.<\/p>\n\n\n\n