They are scaling deployments with more servers and VMs, run high-capacity links, leverage containers, and routinely add new observability, security, and monitoring tools. On top of that, they're often running hundreds or even thousands of apps which, unknown to IT, could include rogue software such as crypto mining or BitTorrent.

With ever-increasing volumes of application-oriented data, it's difficult for IT teams and tools to focus on the most actionable activity and avoid wasting resources processing irrelevant traffic.

Often we inundate security, observability, compliance and network monitoring tools with low-risk, low-value traffic, making them less effective and requiring needless scaling.

Additionally, false positives and alerts can overwhelm NetOps, CloudOps and SecOps teams, obscuring the root causes of network and application performance issues and the real threats buried in volumes of undifferentiated traffic.

'Old school' solutions

Traditionally, IT teams have taken laborious steps to identify applications based on network traffic, by either hardwiring ports to specific applications or writing regular expressions to inspect traffic patterns and identify apps.

Such manual workarounds bring their own challenges. When change occurs, such as growth in an application's usage or the introduction of new applications, NetOps teams must update network segmentation. And app updates can change traffic patterns and behaviour, meaning IT must constantly test and update their homegrown regex signatures. For the cloud, implementing such stopgap measures is difficult, if not impossible.

Until now, it's been hard to isolate cloud traffic by application type and specify whether or not it gets inspected by tools. Visibility has been siloed, and filtering options often only go up to Layer 4 elements, forcing organisations to pass all traffic through their tools or risk missing potential threats.

However, having each tool (intrusion detection system, data loss prevention, advanced threat detection, network analytics, forensics and so on) inspect packets to filter irrelevant traffic is inefficient and costly, as most tool pricing is based on traffic volume and processing load.

While packet brokering can reduce traffic, it requires programming knowledge to maintain complex rules. And although some systems provide a level of application filtering, it's hard to use, identifies a limited number of applications, and doesn't typically share this insight. Further, the filters require ongoing maintenance to keep up with changing application behaviour.

Visualise and filter cloud apps

Application filtering intelligence (AFI) such as my own company's, brings application awareness to multi-cloud environments. The technology automatically extends Layer 7 visibility to identify more than 3,500 common business and network applications traversing the network and lets users select and deliver only high-value or high-risk data based on application, location and activity.

Applications are classified into categories that are automatically updated as the landscape evolves. This allows a team to take actions on a 'family' of applications versus setting policies on individual apps. Examples of application families include antivirus, audio/video, database, ERP, gaming, messenger, peer-to-peer, telephony, webmail, and dozens more.

Now each tool is more efficient, since it no longer needs to store and process large volumes of irrelevant traffic. NetOps can apply existing tools across a larger area by prioritizing only core business applications and accelerate investigation of network and application performance issues with easier data isolation.

SecOps teams can extend current tools to a larger attack surface, securing more of the network and preventing sensitive data, such as personally identifiable information (PII), from being routed to monitoring and recording tools. For more on how to benefit from AFI, check out this paper.

While identifying applications a serious challenge in the cloud, obtaining even basic metadata such as NetFlow, is problematic in public infrastructure as a service (IaaS). It's possible to derive basic details such as which IP addresses are used and by whom, along with port and protocol details.

But the real need is for summarised information, context-aware information about raw packets, based on Layers 4–7, that provides insights into user behaviour, security breaches, customer experience and infrastructure health.

Advanced metadata attributes expand on app layer visibility and support a comprehensive approach to obtaining application behaviour. Especially when deploying workloads in the cloud, users can acquire critical flow details, reduce false positives by separating signal from noise, identify nefarious data extraction, and accelerate threat detection through proactive, real-time traffic monitoring as well as troubleshooting forensics.

Observability and SIEM solutions use this information to correlate and analyse log data from servers and security appliances. Network security and monitoring tools leverage this metadata to deliver the insight and analytics needed to manage the opportunities and risks associated with cloud deployments.

And administrators can automate anomaly detection, stop cyber threats that overcome perimeter or end-point protection, and identify bottlenecks and understand latency issues.

Based on Layers 4–7, application metadata intelligence (AMI) supplies network and security tools with more than 5,000 metadata characteristics that shed light on the application's performance, customer experience, and security. Advanced tech extracts and appends these elements to NetFlow and IPFIX. Records include:

• Identification: Social media user, file and video names, SQL requests
• HTTP: URL identification, command response codes
• DNS parameters: 39 elements, including request/response, queries, and device identifiers
• IMAP and SMTP email-based communications with sender and receiver addresses
• Service identification: Audio, video, chat, and file transfers for VoIP and messaging
• Customer/network awareness: VoIP (SIP, RTP) and mobile (GTP, HTTP/2) control/signalling and user/data plane sessions

Advanced L7 metadata can be applied in a variety of use cases. AMI's principal deployment is in providing metadata to SIEM and observability tools for security analysis. This can help to:

• Identify use of weak ciphers and expired TLS certificates.
• Investigate suspicious network activity by detecting unauthorised remote connections, bandwidth usage, connection longevity, or an unusual quantity of SSH, RDP, or Telnet sessions.
• Detect data exfiltration by monitoring the volume and types of DNS requests implying DNS tunnelling and evaluating the legitimacy of the domains.
• Pinpoint security breach origins with time-window analysis of Kerberos, SMB, and HTTP use to isolate the prior and post protocol activities that lead up to an incident.
• Find suspicious behaviours that suggest compromised credentials or brute force attacks, such as high-privilege user activity, logins from unauthorized systems or multiple hosts, and HTTP client errors.

While IaaS and private cloud orchestration and management platforms are remarkably resilient, dynamic, and infinitely scalable, they don't offer next-generation network packet brokers (NGNPB) with a deep observability pipeline. Such brokers aggregate, filter and distribute all traffic to the proper security and networking tools. They also provide the compute power behind AFI and AMI.