February 24, 2015 By Douglas Bonderud 3 min read

How much of the World Wide Web is readily available to search engines? Chief information security officers and IT security professionals know the amount of data hidden from plain sight is substantial — something akin to an iceberg carrying nine-tenths of its weight below the surface.

However, according to a new Global Commission on Internet Governance report, the number is much larger. Just 0.03 percent of the so-called Deep Web is available to search engines, while the even-deeper Dark Web is deliberately hidden and unavailable when using standard browsers.

Such a massive piece of virtual real estate that is essentially unmonitored by Internet oversight agencies raises the question: Is there any hope for cybersecurity in the dark?

Deep Web Versus Dark Web

To understand the effect of Dark Web data, it is first important to separate the Deep Web from its shadowy counterpart. The report, “The Impact of the Dark Web on Internet Governance and Security,” defines the Deep Web as “a class of content on the Internet that, for various reasons, is not indexed by search engines.” For most major engines, this lack of indexing is tied to profit. While the information is readily available to those who look, so few are interested that actively crawling for this content provides little to no return on investment.

The Dark Web, meanwhile, is “a part of the Deep Web that has been intentionally hidden and is inaccessible through standard Web browsers.” Powered by networks such as TOR and I2P, this hidden Web makes it possible for users to remain entirely anonymous. While in some cases, this anonymity is used simply as a way to protect free speech or for government agencies to keep top-secret data under wraps, there is another side to this darker corner of the Web filled with cybercrime, the transfer of illegal goods and even terrorism. Are Internet governance and cybersecurity even possible in this environment?

Here Comes the Calvary

According to a recent Naked Security article, it’s only a matter of time before law enforcement and other agencies gain some measure of control over the Dark Web. The article likens the existing hidden Web to the Wild West — even though it was once larger than the settled territories of the United States, even this lawless land eventually found itself bound by law and order.

According to the Global Commission on Information Governance report, the following are six key monitoring areas that are essential to the success of any governance effort:

  1. Mapping the Hidden Services Directory: Both TOR and I2P use a distributed hash table system to hide database information. Strategically deployed nodes could monitor and map this network.
  2. Customer Data Monitoring: There will be no monitoring of consumers themselves, but rather destination requests to track down top-level rogue domains.
  3. Social Site Monitoring: This includes watching over popular sites such as Pastebin to find hidden services.
  4. Hidden Service Monitoring: Agencies must “snapshot” new services and sites as they appear for later analysis, since they disappear quickly.
  5. Semantic Analysis: A shared database of hidden site activities and history should be built.
  6. Marketplace Profiling: Sellers, buyers and intermediary agents committing illegal acts should be tracked.

The bottom line for businesses? While the Dark Web does not pose any immediate or obvious threat, it exists nonetheless and operates as a catchall both for users seeking anonymity and those looking to operate outside the law. Monitoring this hidden corner of the Web is by no means impossible. It comes down to the choices nation-states and private companies are willing to make. How much light must be thrown at the Dark Web to make it safe, while still respecting the right to Internet anonymity? Is a known darkness better than none at all?

Image Source: iStock

More from

How will the Merck settlement affect the insurance industry?

3 min read - A major shift in how cyber insurance works started with an attack on the pharmaceutical giant Merck. Or did it start somewhere else?In June 2017, the NotPetya incident hit some 40,000 Merck computers, destroying data and forcing a months-long recovery process. The attack affected thousands of multinational companies, including Mondelēz and Maersk. In total, the malware caused roughly $10 billion in damage.NotPetya malware exploited two Windows vulnerabilities: EternalBlue, a digital skeleton key leaked from the NSA, and Mimikatz, an exploit…

3 Strategies to overcome data security challenges in 2024

3 min read - There are over 17 billion internet-connected devices in the world — and experts expect that number will surge to almost 30 billion by 2030.This rapidly growing digital ecosystem makes it increasingly challenging to protect people’s privacy. Attackers only need to be right once to seize databases of personally identifiable information (PII), including payment card information, addresses, phone numbers and Social Security numbers.In addition to the ever-present cybersecurity threats, data security teams must consider the growing list of data compliance laws…

ICS CERT predictions for 2024: What you need to know

4 min read - As we work through the first quarter of 2024, various sectors are continuously adapting to increasingly complex cybersecurity threats. Sectors like healthcare, finance, energy and transportation are all regularly widening their digital infrastructure, resulting in larger attack surfaces and greater risk exposure.Kaspersky just released their ICS CERT Predictions for this year, outlining the key cybersecurity challenges industrial enterprises will face in the year ahead. The forecasts emphasize the persistent nature of ransomware threats, the increasing prevalence of cosmopolitical hacktivism, insights…

Topic updates

Get email updates and stay ahead of the latest threats to the security landscape, thought leadership and research.
Subscribe today