Skip to main content
For Cybersecurity & IT Solutions Call (310) 955-1600
  • Contact Us
search
  • About Us
    • Message from the President

      Shamrock exists as a trusted ally to enterprise IT leaders responsible for making the most difficult and sensitive decisions related to technology procurement.

      We provide comprehensive and objective analysis at no cost to our customers, helping you make the right decisions on any product from any provider at the guaranteed best price.

      Paul Cooney
      President


      Contact me for a FREE on-site walk through.

      Schedule Now
    • Our Company
    • In The News
      News Press Release

      CIO Review Awarded Shamrock as “20 Most Promising AWS Solution Providers” for 2019

      We are proud to announce that Shamrock has been recognized by CIO Review at one of the “20 Most Promising AWS Solution Providers” for 2019! Check out the article on how our AWS solutions are changing the game for IT decision makers around the globe.
      News Press Release
      Shamrock & Google Come Together to Talk Cloud Computing
      News Press Release
      Insight Success: Shamrock Consulting Group: The 10 Most Innovative Telecom Solution Providers 2018
  • Solutions
      • Cloud Services
        • cdc-wide-v2Cloud Direct Connect

          Connect From Any Location To Any Cloud.

          AWS Direct Connect | Azure ExpressRoute | Google Cloud Interconnect | IBM Cloud Direct Link

          Shamrock can reduce your cloud egress costs by up to 80%

          Learn More
        • cloud-migration-v2Cloud Migration

          Proven leader with successful hands-on cloud deployments, systematically analyzing every aspect of your cloud infrastructure to avoid wasted dollars.

          34% Average Savings on Cloud Services through Shamrock

          Learn More
        • cloud-security-1000×500(2)Cloud Security

          Whether your company is premise-based, in a private cloud, hybrid cloud, or multi-cloud, Shamrock has you covered. We will help you secure and easily monitor your corporate environment in real-time.

          An industry leader in corporate security

          Learn More
        • cloud-cost-optCloud Cost Management

          Shamrock has carefully vetted the top performing tools for public and private cloud orchestration and cost management to service deployments of any size, on any budget at guaranteed best rates.

          Shamrock can reduce your public cloud spend by 40%

          Learn More
        • Platinum Public Cloud Services
          • aws-plat-v2

            Whether you’re looking to migrate to AWS, better manage your current AWS environment, or expand your capabilities within AWS, Shamrock can help.

            We’re top rated AWS consultants for a reason – because we’re a team of vendor-neutral, award-winning cloud experts who always act in the best interests of our clients, and we always guarantee the best price on AWS products and services.

            Learn More
          • azure-plat

            We’ll help you maximize your Azure experience and take your business to the next level by designing, deploying and managing scalable infrastructure and application-based solutions. We’ll even facilitate your migration from your on-prem or environment or from another cloud provider to Azure, in whatever capacity you choose.

            Shamrock is a Certified MS Azure Partner, offering expert consulting services at the guaranteed best price.

            Learn More
          • gcp-plat

            Google’s Partner Network is one of the most prestigious in the industry, and Shamrock is honored to be a part of it. As a Google Partner, Shamrock is your go-to resource for support in GCP. We provide free 24/7 US-based support, migration support, scalable solutions, cloud cost management and more within GCP, all for the guaranteed best price.

            Shamrock is the market leader in GCP cost reduction, GCP migration services, GCP Interconnect direct connects and much more.

            Learn More
      • Data Center
        • cloud-security-1000×500(2)Cloud Security

          Whether your company is premise-based, in a private cloud, hybrid cloud, or multi-cloud, Shamrock has you covered. We’ll help you secure and easily monitor your environment in real-time to ensure proactive protection of your corporate IT environment. Join our growing list of Fortune 500, Fortune 100 and Fortune 50 customers and see why Shamrock is an industry leader in corporate security.

          Join our growing list of Fortune 500 and see why Shamrock is an industry leader in corporate security.

          Learn More
        • data-center-colocation-v2(1)Colocation

          We’ve been recognized as an industry leader in data center sourcing and procurement on a global scale. We represent every major data center operator both domestically and globally, so no matter the size, scope or area of your needs, we’ve got you covered. With over 1,200 major networks, enterprises, and business partners available for interconnection and traffic exchange via direct cross connects, peering and Ethernet services

          We Sell Data Center Space!

          Learn More
        • cdc-wide-v2Cloud Direct Connect

          Shamrock is an industry leader in provisioning cloud direct connects from any business or data center location, anywhere in the world, at any speed (10 Mbps to 100Gbps).

          We’re also the exclusive solution provider for 100G cloud direct connects into Google Cloud Interconnect and AWS Direct Connect.

          Join Out 100GIG Pilot Program.

          Learn More
      • Telecom Solutions
        • tem-squareTelecom Expense Management (TEM)

          Our portfolio of services includes business process outsourcing (BPO), telecommunication audit, procurement advisory, and wireless expense management. We strive to produce sustained results for your organization.

          Learn More
        • ent-mob-squareEnterprise Mobility Management

          Shamrock’s mobile cost management experts can help you drastically reduce your enterprise mobility costs without any provider or contract changes. Our technology uses patented algorithms and machine learning. The result: instant and tangible savings!

          Learn More
        • art1-new-1-smallVoice Services

          Shamrock Consulting provides Long Distance Services with cost-effective in-state, state-to-state, international and toll-free calling for businesses. International or Domestic Toll-Free Numbers, Conferencing Services, powerful cost-management options.

          Learn More
        • ucaas-thumb-optUCaaS / Hosted PBX

          As industry-leading UCaaS consultants, Shamrock has negotiated direct partnerships with every major UCaaS provider to ensure that our clients receive the best possible solution at the best possible price. Guaranteed.

          Learn More
        • 3-new-smallAnalog Line Consolidation

          Dynamically manage traffic pattern changes and get high-quality, reliable voice services that leverage your existing PBX investment. Pbx Service Providers, SIP Service, Pbx Pricing, Cloud Pbx and more.

          34% Average Cost Savings with Shamrock

          Learn More
      • WAN Solutions
        • 2-new-1Broadband Aggregation

          The need to future proof your network has created an unprecedented demand for fiber to the premise. Dark Fiber, Fiber Ethernet, DS3, OC-x, Fixed Wireless (Microwave, lazer, WIMAX), Ethernet over Copper (EOC), T1, NxT1, Cable, DS, Shamrock Consulting Group has all of your options in one place.

          Let's Start With Understanding Your Internet Access Options!

          Learn More
        • art2-1-new-smallInternet Access

          The need to future proof your network has created an unprecedented demand for fiber to the premise. Dark Fiber, Fiber Ethernet, DS3, OC-x, Fixed Wireless (Microwave, lazer, WIMAX), Ethernet over Copper (EOC), T1, NxT1, Cable, DS, Shamrock Consulting Group has all of your options in one place.

          Let's Start With Understanding Your Internet Access Options!

          Learn More
        • art8-new-smallPrivate WAN (MPLS / VPLS)

          Global enterprises, financial institutions, the largest networks and the Internet’s foremost content companies trust us with what matters most to them—their information assets. Shamrock Consulting Group can design wide area networks for any size, any budget, anywhere.

          Let's Start With Understanding Your WAN Objectives!

          Learn More
        • art3-new-smallDark Fiber

          We partner directly with every major Dark Fiber providers in the U.S., Europe and Asia-Pacific. We also work with cities and municipalities such as Los Angeles County, Orange County, the San Francisco Bay Area, New York City and elsewhere to find you the best possible solution at the guaranteed best price.

          Looking for fiber providers in your area?

          Learn More
        • art7-new-smallSD-WAN

          As top-rated SD WAN consultants who partner directly with every leading SD WAN provider in the industry, Shamrock is committed to helping you make well-informed procurement decisions at the guaranteed best price. We offer a free consultation to identify the best solution for your needs and budget.

          Considering an SD-WAN solution for your company?

          Learn More
      • Cybersecurity
        • crowd-strikeCrowdstrike EDR/MDR

          As a CrowdStrike partner, the Shamrock team as a whole are big fans of the company’s cloud native endpoint protection platform. But don’t take our word for it – we’ve had hundreds of customers using CrowdStrike tell us how much they love the platform, and many of them have said that they have no idea how they ever lived without it.

          Licensing discounts of up to 50 percent below market rates.

          Learn More
        • penntesting-v2Penetration Testing

          We’ve formed deep partnerships with the most trustworthy cybersecurity vendors in the industry, and we guarantee the best price from every single one of them. Black, White & Gray Box Testing, Cloud Pentesting, Mobile & Web App Pentesting, Wireless Pentesting, Physical Pentesting, Client-Side Pentesting. In addition to pentesting, we can also help you out with next-generation firewalls, endpoint protection and corporate email security.
          Schedule Your Free Pentest Analysis.

          Learn More
        • vulnerability-testingVulnerability Management

          We take a look at some great security solutions that achieve real time visibility into all aspects of a corporate network, even reaching the most remotely deployed endpoints. If you’re interested (and you probably should be), Shamrock Consulting Group can secure these cutting-edge tools for you at the best prices.

          We can offer the best prices on CrowdStrike products and can also provide a free security assessment.

          Learn More
        • ngfwNGFW & Software Defined Perimeter

          How can companies be both agile and secure in these unprecedented times? For help with empowering your WFH teams with the tools and processes they need, speak to a Shamrock consultant today.

          Our strategic partners are comprised of the very best in the industry from a security standpoint (Trustwave, AlertLogic, CrowdStrike, Fortinet, PaloAlto, Carbon Black, etc.).

          Learn More
        • cloud-security-smallCloud Security

          Whether your company is premise-based, in a private cloud, hybrid cloud, or multi-cloud, Shamrock has you covered. We’ll help you secure and easily monitor your environment in real-time to ensure proactive protection of your corporate IT environment.

          Join our growing list of Fortune 500 and see why Shamrock is an industry leader in corporate security.

          Learn More
  • RFP Services
  • Industries
    • biotech-telecom-consulting-v2Biotech

      Safeguard Proprietary, Mission Critical Data Whether its R&D for a world changing innovation or straightforward emails to accounting, Shamrock provides a comprehensive suite of security services – including Managed Firewall, Intrusion Detection/Prevention, and Vulnerability Scanning – to ensure that your organization’s data is safeguarded and kept out of the wrong hands.

      Let's Start With Understanding Your Biotech Telecom Objectives!

      Learn More
    • real-estate-construction-cloud-solutionsConstruction & Development

      Whether you are looking to increase network uptime, reduce operational expenses, or build a next generation global development company, Shamrock Consulting Group has turnkey solutions tailored to the needs of the Construction and Development industry at guaranteed best rates.

      Let's Start With Understanding Your Telecom Objectives for Construction & Development sites!

      Learn More
    • finance-cloud-solutionsFinance

      Shamrock Consulting Group’s Financial Services practice offers customers robust custom architected network, cloud, data center, and unified communications solutions that provide the security, compliance, and uptime their organizations require to function on a day-to-day basis.

      Let's Start With Understanding Your Financial Organization's Telecom Objectives!

      Learn More
    • healthcare-cloud-solutionsHealthcare

      Shamrock has cross-disciplinary knowledge of healthcare-specific IT apps. Shamrock has extensive rural network design and build experience. In addition to 100+ partners, Shamrock has experience, connections and industry tools to identify, design to, spec, procure and project manage complex multi-vendor deployments. Healthcare institutions, financial institutions, the largest networks and the Internet’s foremost content companies trust us with what matters most to them—their information assets.

      Let's Start With Understanding Your Healthcare Telecom Objectives!

      Learn More
    • media-entertainment-cloud-solutionsMedia & Entertainment

      Whether you are looking to increase network uptime, reduce operational expenses, or build a next-generation global studio, Shamrock Consulting Group has turnkey solutions tailored to the needs of the Media and Entertainment industry at guaranteed best rates. Nobody delivers more connectivity solutions to Media and Entertainment companies than Shamrock Consulting Group!

      300+ Media and Entertainment Customers Served!

      Learn More
    • retail-cloud-solutionsRetail / Restaurant

      Shamrock Consulting Group offers its retail customers the best-fit, access technologies at each of their sites to ensure complete coverage and uptime at low costs. We design solutions that provide PCI-compliant security for credit card transactions and support for data communications for key retail applications—including point-of-sale, back-office systems, training, and digital video surveillance.

      Let's Start With Understanding Your Retail Business Operational Objectives!

      Learn More
    • saas-cloud-solutionsSAAS Providers

      Whether you are looking to increase network uptime, reduce operational expenses, or build a next generation global Software as a Service Platform, Shamrock Consulting Group has turnkey solutions tailored to the needs of the SaaS Industry at guaranteed best rates. Low, Mid, and High-density data center options with robust connectivity options to extend service coverage all over the globe.

      Let's Start With Understanding Your SAAS Objectives!

      Learn More
  • Tools
  • Press
  • Contact Us

How Data Lakes are Advancing Threat Hunting

Data lake threat hunting

Businesses that can harness the capacity of data lakes are the ones who truly begin to experience the benefits of big data. One of the areas where this could make a significant difference is in cybersecurity, with specific regards to advanced threat hunting.

In this article, we’ll look at how data lakes differ from other data repositories (i.e. data warehouses), what challenges threat hunters currently face and how the next-gen SIEM, together with data lakes, is advancing the field.

We’ll also explain why seeking guidance from an experienced 3rd party like Shamrock Consulting Group can give your business a valuable leg up before diving into the depths of the data lake.

What is a Data Lake, Anyway?

One way to imagine a data lake is to think of a huge body of water fed by lots of incoming streams. Bring the cloud into the picture and you can imagine data flowing down from the heavens into that one big data source.

Unlike a data warehouse, which stores data hierarchically in files and folders, a data lake is a repository for raw data. This data is assigned a unique identifier and tagged with an extended set of metadata tags.

Although the term ‘data lake’ is intertwined with the Hadoop framework, its usage has now been extended to cover any data storage facility where the purpose of storage is not pre-defined.

A modern data lake can take in data from millions of logs and events every second. This includes new sources of data from click-streams, social media, mobile apps and IoT devices. Utilizing this data is the challenge of our age, not just for threat hunters but also for data analysts connected to various areas of the business.

The value of the data lake has already been proven by research. For example, in 2017, an Aberdeen Group research project for AWS found that businesses that invested in data lake technology outperformed their competitors by 9%.

The evolution of the SIEM (which we will go into later), means that whereas previously threat hunters would have had to tap into numerous data streams individually, they can now access the data lake as a whole. Pretty sweet, right?

The Challenges of Threat Hunting Today

Cyber-attacks are persistent and come in a variety of forms. Threats to enterprise networks can be broadly broken down into three groups:

Malicious Insiders

These are people who work within the business and use their access privileges to cause damage to or defraud the organization that employs them. Threat hunters use browser forensics and study unusual authentication patterns to identify planned or actual attacks.

Data Exfiltration

This is where a successful attack has previously occurred and sensitive data is being illicitly diverted outside of the company network. Anomalies in the frequency and/or size of data transfers are red flags for this type of attack.

External Attacks

These are ongoing attacks which have yet to breach the network. They include advanced persistent threats (APTs), denial-of-service (DDoS) attacks and zero-day exploits. Threat hunters need to be alert to both known attack signatures and the symptoms of new types of attack.

As networks become more complex, security teams are faced with the task of making sense of inputs from a huge number of devices and systems, including cloud-based software, apps and services. For starters, there are logs from firewalls, VPNs, DLP tools, proxies, servers and endpoints. Then there are multiple communications from devices using different protocols such as DHCP, IP, TCP, HTTP and FTP.

Human threat hunters would be simply unable to deal with this data flood without help from sophisticated security information and event management (SIEM) systems.

The SIEM: Past and Present

A data lake is worthless to a threat hunter unless he or she can query the data within it. To understand how data lakes are advancing threat hunting, we have to also look at how SIEMs have evolved over the past twenty years.

SIEMs were the natural next step forward from separate security information management (SIM) and security event management (SEM) systems commonly in use at the beginning of the 21st century. These were only able to cope with small-scale data and had poor data visualization capability. Human operators had to manually ingest the data, write correlation rules and analyze the outputs. It was a slow process, but cyber-attacks at this time were relatively predictable and easy to spot.

Although SIEMs had been around for a decade by 2010, it was only then that they started becoming commonplace in businesses. SIEMs could automatically integrate logs from different devices and systems in one place, enabling threat hunters to detect more subtle threat actors (i.e. those that could move laterally within a breached network). However, analysis was still performed manually on a sub-set of filtered data using limited pre-built visualization tools.

SIEM 3.0

The next generation SIEM tech, sometimes referred to as SIEM 3.0, brought machine learning (ML) into the equation. This smart system not only ingests data automatically but also makes decisions based on pattern detection capabilities that go way beyond human capabilities. For common threats, SIEM 3.0 can trigger Security Orchestration, Automation and Response (SOAR) tools to take remedial action without human intervention.

For more complex threats, the next-gen SIEM can alert a human threat hunter, who can then attempt to determine the source and nature of the threat. All security information can be accessed via a single pane of glass through which threat hunters receive a high-fidelity, full context view of security events.

Data can be subjected to a range of sophisticated analytic techniques such as user and entity behavior analytics (UEBA), which matches observed behavior with expected behavior and picks up on any anomalies. There is no need for time-consuming IT requests to build an analytics platform because this is already pre-built into the system.

Unlike earlier iterations, security professionals can query as much of the historical and current data as can be feasibly stored. With data lake integration, that figure is practically limitless.

Querying the Data Lake

So now we’ve come full circle to the data lake. The unstructured nature of data in a lake makes it suitable for a wide range of different queries including SQL queries, full text search, real-time analytics and machine learning. Integrated with advanced SIEM tech, searching for correlations and understanding the full context of any patterns becomes as simple as clicking a search button on a central user interface. This makes for a faster kill-chain which, as any threat hunter knows, is a very satisfying outcome – not to mention great news for the business itself.

The virtually unlimited capacity of data lakes means that threat hunters can query vast quantities of historic and present data. This makes it much easier to discover subtle anomalies when investigating previous attacks, and the results can be fed back to the SIEM to help it improve its ability to recognize and derail future attacks.

Before You Take a Dip in the Data Lake…

Data lakes are available through AWS (S3), Azure Data Lake and Apache Hadoop among others and, as with other distributed technologies, your choice of data lake and SIEM is critical to the security function of your business.

Although data lakes have advantages over data warehouses, most businesses will need both since warehouses are optimized for fast queries connected to line of business applications. Shamrock can help ensure that you make the right choices for your business every step of the way.

Our team has over a decade of experience working alongside Fortune 500, Fortune 100 and Fortune 50 companies on their security solutions and cloud connectivity. As a vendor-neutral source of specialist support, you can trust Shamrock to guide you in the most profitable direction for your business (and our security analysis is free!).

Ben Ferguson

Ben Ferguson is the Vice President and Senior Network Architect for Shamrock Consulting Group, an industry leader in digital transformation solutions. Since his departure from Biochemical research in 2004, Ben has built core competencies around cloud direct connects and cloud cost reduction, enterprise wide area network architecture, high density data center deployments, cybersecurity and Voice over IP telephony. Ben has designed hundreds of complex networks for some of the largest companies in the world and he’s helped Shamrock become a top partner of the 3 largest public cloud platforms for AWS, Azure and GCP consulting. When he takes the occasional break from designing networks, he enjoys surfing, golf, working out, trying new restaurants and spending time with his wife, Linsey, his son, Weston and his dog, Hamilton.

best-price-guarantee-tick
Learn About Our Best Price Guarantee