New blog articles in Microsoft Community Hub

Check This Out! (CTO!) Guide (April 2024)

BrandonWilson — Tue, 28 May 2024 04:16:34 GMT

Hi everyone! Brandon Wilson here once again with this month’s “Check This Out!” (CTO!) guide.

These posts are only intended to be your guide, to lead you to some content of interest, and are just a way we are trying to help our readers a bit more, whether that is learning, troubleshooting, or just finding new content sources! We will give you a bit of a taste of the blog content itself, provide you a way to get to the source content directly, and help to introduce you to some other blogs you may not be aware of that you might find helpful.

From all of us on the Core Infrastructure and Security Tech Community blog team, thanks for your continued reading and support!

Title: Speaking in Ciphers and other Enigmatic tongues fresh content update!

Source: Ask the Directory Services Team

Author: Jim Tierney

Publication Date: 4/3/24

Content excerpt:

So, your company purchases this new super awesome vulnerability and compliance management software suite, and they just ran a scan on your Windows Server 2008 domain controllers and lo! The software reports back that you have weak ciphers enabled, highlighted in RED, flashing, with that "you have failed" font, and including a link to the following Microsoft documentation…

Title: NTLM vs Kerberos

Source: Ask the Directory Services Team

Author: Josh Mora

Publication Date: 4/23/24

Content excerpt:

In this post, we will go through the basics of NTLM and Kerberos. We will explain using the three Ws, covering what the main differences between them are, how to identify when a protocol is being used over the other, and why one is safer than the other.

Title: Coming Soon: Transition to WS2012 ESUs enabled by Azure Arc

Source: Azure Arc

Author: Aurnov Chattopadhyay

Publication Date: 4/3/24

Content excerpt:

Announced during last week’s Windows Server Summit 2024, customers will be able to transition to Azure Arc for WS2012 Extended Security Updates enabled by Azure Arc for Year 2 if they had purchased Year 1 of Extended Security Updates for Volume Licensing. This experience will ask customers to supply their Invoice Id, corresponding to WS2012 ESU purchases from Volume Licensing.

With its pay as you go flexibility as an Azure billed service and inclusion of Azure management services like Azure Update Manager and Machine Configuration at no additional cost, WS2012 ESUs enabled by Azure Arc offer financial and experience benefits to customers. This is especially critical to support customers in their journey of migrating and modernizing End of Life infrastructure to Azure.

Title: Public Preview of Azure Arc Site Manager

Source: Azure Arc

Author: Nathan Parikh

Publication Date: 4/22/24

Content excerpt:

This week we are excited to announce the Public Preview of Azure Arc site manager. We designed site manager to meet the needs of customers who manage solutions on the adaptive cloud and want to view and monitor their resources according to their physical locations, such as stores, restaurants, and factories. Within site manager, customers can create Arc sites to represent their on-premises environments and see centralized monitoring information across their edge infrastructure.

Title: Cloud Governance Guidance in Microsoft's Cloud Adoption FrameworkCloud Governance Guidance in Microsoft's Cloud Adoption Framework

Source: Azure Architecture

Author: Stephen Sumner

Publication Date: 4/8/24

Content excerpt:

We are thrilled to announce the latest enhancement to Microsoft's Cloud Adoption Framework for Azure. We comprehensively updated our cloud governance guidance in the Govern section of the Cloud Adoption Framework (CAF). The updated governance guidance represents Microsoft's commitment to supporting your organization's cloud journey, offering a clearer, more accessible, and comprehensive path to effective cloud governance. It encompasses identity, cost, resource, data, and AI governance among other areas of governance categories.

Whether you're a startup looking to scale efficiently or a large enterprise aiming to refine your governance practices, we designed this governance guidance to meet your needs and guide you to where you need to be.

Title: On Demand Capacity Reservation is now available in Azure for US Government

Source: Azure Compute

Author: Megan Pennie

Publication Date: 4/29/24

Content excerpt:

Today, we're announcing the general availability of on demand capacity reservations for Azure Virtual Machines in Azure for US Government Cloud . You can now manage and reserve capacity with guaranteed SLA for VM sizes available on Azure for US Government Cloud.

As part of our ongoing commitment to expanding our global reach and providing On Demand Capacity Reservation benefits to our customers, we have diligently worked hard for months to bring this feature to Azure for US Government Cloud.

Title: Azure billing meters: What you need to know about the upcoming changes

Source: Azure Governance and Management

Author: Vahe Minasyan

Publication Date: 4/8/24

Content excerpt:

This article is to inform you of some important changes that we are making as part of our ongoing efforts to improve your Azure experience and increase pricing transparency. We are making changes to the billing meters that could affect some Azure services and resources that you may use, and we want to make sure you understand what the changes are, why we are doing them, and how they will impact your billing.

These updates will not impact your prices, but you may notice some changes in how your Azure consumption is shown on your invoice, price sheet, API, and other Cost Management tools.

Title: Perform bulk NSG rule rollout across multiple target NSGs

Source: Azure Infrastructure

Author: Sourav Bera

Publication Date: 4/22/24

Content excerpt:

Network Security Groups (NSGs) are a fundamental aspect of Azure networking, providing a layer of security to control traffic flow within virtual networks. However, managing NSG rules across multiple NSGs can be a daunting task, especially when done manually. This article introduces a powerful PowerShell script that allows you to perform bulk NSG rule rollouts across multiple target NSGs, saving you time and ensuring consistency across your network.

Title: Enhancing Azure Connectivity: Sharing PaaS instance across customer tenants on Azure

Source: Azure Infrastructure

Author: Aquib Qureshi

Publication Date: 4/28/24

Content excerpt:

I’ve come across a scenario where one of my customer using Azure SQL DB wanted to share their Database with other customer who was also hosted on Azure. They were struggling to establish site-to-site connectivity so that Customer B could access Customer A’s network, enabling them to connect to the Azure SQL DB via the site-to-site tunnel. Though this can be achieved, there are better ways to connect to Azure SQL DB, or any PaaS instance for that matter, with another customer who is using Azure. This can also be used by customers who have multiple Azure AD tenants.

Title: VMware HCX Troubleshooting with Azure VMware Solution

Source: Azure Migration and Modernization

Author: Rene van den Bedem

Publication Date: 4/8/24

Content excerpt:

VMware HCX is one of the Azure VMware Solution components that generates a large number of service requests from our customers. The Azure VMware Solution product group has worked to cover the most common troubleshooting considerations that you should know about when using VMware HCX with the Azure VMware Solution.

Title: Microsoft Azure ExpressRoute Overview Cheat Sheet

Source: Azure Networking

Author: Gene Whitaker

Publication Date: 4/9/24

Content excerpt:

Microsoft Azure ExpressRoute lets you extend your on-premises networks into the Microsoft Cloud over a private connection with the help of a connectivity provider. With ExpressRoute, you can establish connections to Microsoft Cloud services, such as Microsoft Azure and Microsoft 365.

Use this ExpressRoute Overview Cheat Sheet to quickly learn and gain access to the following information:

ExpressRoute Main Components
Key Benefits and Features
The Different Connectivity Models
Various Helpful Links
Diagram demonstrating ExpressRoute circuits along with features such as Global Reach and FastPath

This cheat sheet is compiled with important and most common information to learn ExpressRoute.

Title: Network traffic observability with virtual network flow logs

Source: Azure Networking

Author: HarshaCS

Publication Date: 4/24/24

Content excerpt:

Azure Network Watcher provides network monitoring and troubleshooting capabilities to increase observability and actionable insights with out-of-box health metrics & topology visualization, connectivity monitoring, traffic monitoring and diagnostics suite. For on-premises workloads, network administrators rely on NetFlow or IPFIX to address these use cases. Virtual network flow logs are a capability of Network Watcher service to address these scenarios for Azure and hybrid networks and we are excited to announce that virtual network flow logs are now transitioning from public preview to general availability.

Title: Understanding the core concept and routing of vWAN with Example

Source: Azure Networking

Author: Sourav Das

Publication Date: 4/26/24

Content excerpt:

What is virtual WAN? Azure Virtual WAN is a NAAS (networking as a service) to enable simplified global transit networking architecture that brings many networking, security, and routing functionalities together to provide a single operational interface…

Title: New and improved network topology experience in Network Watcher and Azure Monitor Network Insights

Source: Azure Networking

Author: Sagar Gupta

Publication Date: 4/28/24

Content excerpt:

Azure Network Watcher provides network monitoring and troubleshooting capabilities to increase observability and actionable insights. Network Watcher supports four main scenarios: Connectivity Monitoring detects packet loss and latency, built-in health metrics and topology visualization help to locate issues, traffic monitoring tracks network communication pattern, and diagnostics suite enables troubleshooting.

Title: Apply critical update for Azure Stack HCI VMs to maintain Azure verification

Source: Azure Stack

Author: Kimberly Lam

Publication Date: 4/18/24

Content excerpt:

Azure verification for VMs on Azure Stack HCI makes it possible for Azure-exclusive benefits to work outside of the cloud and in on-premises and edge environments. These benefits include Azure Virtual Desktop for Azure Stack HCI, Windows Server Datacenter: Azure Edition, Extended Security Updates (ESUs) for SQL and Windows Server Virtual Machines (VMs) on Azure Stack HCI, and Azure Policy guest configuration. To keep these workloads continuing to function, periodic updates are required to maintain their security and functionality.

Title: Sneak peek at new Azure edge infrastructure at Hannover Messe 2024

Source: Azure Stack

Author: Cosmos Darwin

Publication Date: 4/22/24

Content excerpt:

This week is Hannover Messe 2024, the world’s biggest industrial trade fair. Microsoft is there, including members of the Azure Stack team, showcasing how the Microsoft Cloud enables end-to-end manufacturing solutions that help securely connect people, assets, and business processes, empowering organizations to be more resilient.
Near the center of our booth, you can watch a robotic assembly line put together battery parts. The line features standard OT assets from our partner Rockwell Automation and shows how an adaptive cloud approach together with open standards like OPC UA can accelerate industrial transformation. Azure IoT Operations enabled by Azure Arc flows data from the production line into Microsoft Fabric, enabling real-time monitoring and analysis in the cloud.
And if you look closer, you may spot an exciting new infrastructure solution hosting it all…

Title: Azure Virtual Desktop for Azure Stack HCI now has autoscale

Source: Azure Virtual Desktop

Author: Jessie Duan

Publication Date: 4/11/24

Content excerpt:

On February 2024, we announced the general availability of Azure Virtual Desktop for Azure Stack HCI, which extends the capabilities of the Microsoft Cloud to your datacenters and edge locations. Today, we’re happy to announce that autoscale support on Azure Virtual Desktop for Azure Stack HCI is now in public preview. With the Azure Virtual Desktop autoscale feature, organizations running virtualized desktops and apps on-premises, at the edge or in their datacenter, can optimize costs by turning off idle Azure Virtual Desktop session hosts running on Azure Stack HCI.

Title: Get Azure Reservations and Savings Plans Insights with the Azure Optimization Engine

Source: Core Infrastructure and Security

Author: Helder Pinto

Publication Date: 4/1/24

Content excerpt:

Azure Reservations and Savings Plans commitments have been adopted by many customers with a predictable and steady Azure consumption to achieve considerable savings over on-demand prices. Depending on your on-demand Azure Compute consumption patterns, you may choose one over the other, or even have both working in tandem…

Title: ConfigMgr: Avoiding Remote Management Point Pitfalls

Source: Core Infrastructure and Security

Author: Pavel Yurenev

Publication Date: 4/4/24

Content excerpt:

I’m Pavel Yurenev, a Support Escalation Engineer specializing in Microsoft Configuration Manager at Microsoft Customer Service & Support (CSS). As Reactive Support, we assist customers with issues arising from Microsoft software products. Unfortunately, some supported product configurations are poorly supportable.

Today, I want to discuss certain design features of Configuration Manager Management Points (MPs) and provide guidance for architects and administrators.

Title: Proxies, proxies everywhere but still no Internet. Overview of the Windows Proxies

Source: Core Infrastructure and Security

Author: Will Aftring

Publication Date: 4/8/24

Content excerpt:

Howdy everyone, a quick tangent from our regularly scheduled Introduction to Network Trace Analysis series to talk about the Windows Proxy ecosystem. A Windows Proxy configuration can be a little tricky, so I wanted to add clarity for configuration methods. Scoping things a bit here I will also only be referring to 64-bit applications.

But first, let’s explain what I mean when I say proxy…

Title: Active Directory Hardening Series - Part 4 – Enforcing AES for Kerberos

Source: Core Infrastructure and Security

Author: Jerry Devore

Publication Date: 4/15/24

Content excerpt:

Hi everyone, Jerry Devore here again with another installment in my series on Active Directory hardening. This time I want to revisit a topic I previously wrote about in September of 2020 which is enforcing AES for Kerberos. Since I wrote that blog post a few new tips have come my way. Before we dive in here is a quick re-cap of what was previously discussed…

Title: Controlling AKS egress using an HTTP Proxy

Source: Core Infrastructure and Security

Author: Houssem Dellai

Publication Date: 4/22/24

Content excerpt:

Azure Kubernetes Service (AKS) clusters, whether deployed into a managed or custom virtual network, have certain outbound dependencies necessary to function properly. Previously, in environments requiring internet access to be routed through HTTP proxies, this was a problem. Nodes had no way of bootstrapping the configuration, environment variables, and certificates necessary to access internet services.

This feature adds HTTP proxy support to AKS clusters, exposing a straightforward interface that cluster operators can use to secure AKS-required network traffic in proxy-dependent environments.

Both AKS nodes and Pods will be configured to use the HTTP proxy. Here is an architecture diagram showing the different components.

Title: Building Your Own Copilot for Credit Card Selection

Source: Core Infrastructure and Security

Author: Felipe Binotto

Publication Date: 4/28/24

Content excerpt:

Have you ever found yourself lost in the maze of credit card options, navigating through countless comparison websites, unsure of the accuracy and timeliness of their information? I certainly have. Recently, I embarked on a quest to find the perfect credit card, one that rewards my spending habits with frequent flyer points. However, relying solely on popular comparison platforms left me questioning the reliability of their data, often overshadowed by biased advertisements.

But then, a beacon of hope emerged: the realization that all bank product information is accessible through a common API. With this revelation, I set out to craft my own solution – a personalized Copilot to guide me through the sea of credit card offerings.

Title: Cost allocation is imperative for cloud resource optimization

Source: FinOps

Author: Antonio Ortoll

Publication Date: 4/15/24

Content excerpt:

Like most enterprises, you are probably managing your Azure resources and services centrally and need a way to distribute and “showback” or reallocate the cost of services back to the organizational units that use those services.

Sharing cloud services provides greater flexibility and scalability when they can be dynamically allocated. But as your cloud adoption grows, the quantity of billing and usage data and the speed at which it is delivered can make allocation and reporting a challenge without a strategy and a system to efficiently assign the costs of cloud resources to their specific users.

Title: 7 steps for a successful Azure migration

Source: ITOps Talk

Author: Sonia Cuff

Publication Date: 4/9/24

Content excerpt:

Migrating an on-premises environment to Azure requires preparation, planning, and time. Join Microsoft MVP Gregor Reimling and learn seven key steps for a successful Azure migration.

In this video, you will learn…

Title: Wired for Hybrid - What's New in Azure Networking - April 2024 edition

Source: ITOps Talk

Author: Pierre Roman

Publication Date: 4/22/24

Content excerpt:

Azure Networking is the foundation of your infrastructure in Azure. Each month we bring you an update on What’s new in Azure Networking.

In this blog post, we’ll cover what's new with Azure Networking in April 2024. In this blog post, we will cover the following announcements and how they can help you.

Title: Dual-Region Azure VMware Solution design with Global Reach, using Secure Virtual WAN

Source: ITOps Talk

Author: Jason Medina

Publication Date: 4/24/24

Content excerpt:

This article describes the best practices for connectivity, traffic flows, and high availability of dual-region Azure VMware Solution when using Azure Secure Virtual WAN with Routing Intent and Global Reach. This article breaks down Virtual WAN with Routing Intent topology from the perspective of Azure VMware Solution private clouds, on-premises sites, and Azure native. The implementation and configuration of Secure Virtual WAN with Routing Intent are beyond the scope and are not discussed in this document.

Title: Important update: Deprecation of Azure AD PowerShell and MSOnline PowerShell modules

Source: Microsoft Entra (Azure AD)

Author: Kristopher Bash

Publication Date: 4/1/24

Content excerpt:

In 2021, we described our plans to invest in Microsoft Graph PowerShell SDK as the PowerShell provider for Microsoft Entra and transition away from Azure AD and MSOnline PowerShell modules. In 2023, we announced that the deprecation of Azure AD and MSOnline PowerShell modules would occur on March 30, 2024. We’ve since made substantial progress closing remaining parity gaps in Microsoft Graph PowerShell SDK, and as of March 30, 2024, these PowerShell modules are now deprecated:

Azure AD PowerShell (AzureAD)
Azure AD PowerShell Preview (AzureADPreview)
MS Online (MSOnline)

You should migrate your scripts to Microsoft Graph PowerShell SDK as soon as possible. Information about the retirement of these modules can be found below.

Title: What's new in Microsoft Entra

Source: Microsoft Entra (Azure AD)

Author: Shobhit Sahay

Publication Date: 4/1/24

Content excerpt:

With the ever-increasing sophistication of cyber-attacks, the increasing use of cloud-based services, and the proliferation of mobile devices, it’s essential that organizations secure access for both human and non-human identities to all on-premises and cloud resources, while working continuously to improve their security posture.

Today, we’re sharing feature release information for January – March 2024, and first quarter change announcements. We also communicate these via release notes, email, and the Microsoft Entra admin center.

Title: Introducing new and upcoming Entra Recommendations to enhance security and productivity

Source: Microsoft Entra (Azure AD)

Author: Shobhit Sahay

Publication Date: 4/2/24

Content excerpt:

Managing the myriad settings and resources within your tenant can be daunting. In an era of escalating security risks and an unprecedented global threat landscape, organizations seek trusted guidance to enhance their security posture That’s why we introduced Microsoft Entra Recommendations to diligently monitor your tenant’s status, ensuring it remains secure and healthy. Moreover, they empower you to extract maximum value from the features offered by Microsoft Entra ID. Since the launch of Microsoft Entra recommendations, thousands of customers have adopted these recommendations and resolved millions of resources.

Title: Introducing "What's New" in Microsoft Entra

Source: Microsoft Entra (Azure AD)

Author: Shobhit Sahay

Publication Date: 4/15/24

Content excerpt:

Today, I’m thrilled to announce the public preview of What’s New in Microsoft Entra. This new hub in the Microsoft Entra admin center offers you a centralized view of our roadmap and change announcements across the Microsoft Entra identity and network access portfolio. In this article, I’ll show you how admins can get the most from what’s new to stay informed about Entra product updates and actionable insights.

Title: Enforce least privilege for Entra ID company branding with the new organizational branding role

Source: Microsoft Entra (Azure AD)

Author: James Mantu

Publication Date: 4/18/24

Content excerpt:

I’m pleased to announce General Availability (GA) of the organizational branding role for Microsoft Entra ID company branding.

This new role is part of our ongoing efforts to implement Zero Trust network access by enforcing the principle of least privilege for users when customizing their authentication user experience (UX) via Entra ID company branding.

Previously, users wanting to configure Entra ID company branding required the Global Admin role. This role, though, has sweeping privileges beyond what’s necessary for configuring Entra ID company branding.

Title: Onboard to Azure Arc with Security in Mind

Source: Security, Compliance, and Identity

Author: Simone Oor

Publication Date: 4/17/24

Content excerpt:

Azure Arc allows certain on-premises resources, typically servers, to be managed from Azure, depending on the configuration mode selected and currently available features.

While this allows for a more integrated approach to hybrid environments, it also further blurs the administrative boundary between on-premises and cloud.

This increases the risk that a vulnerability on either side lowers the level of security across the entire plane. This article contains tips for managing this risk and approaching Arc Onboarding with security in mind.

Previous CTO! Guides:

CIS Tech Community-Check This Out! (CTO!) Guides

Additional resources:

Azure documentation
Azure pricing calculator (VERY handy!)
Microsoft Azure Well-Architected Framework
Microsoft Cloud Adoption Framework
Windows Server documentation
Windows client documentation for IT Pros
PowerShell documentation
Core Infrastructure and Security blog
Microsoft Tech Community blogs
Microsoft technical documentation (Microsoft Docs)
Sysinternals blog
Microsoft Learn
Microsoft Support (Knowledge Base)
Microsoft Archived Content (MSDN/TechNet blogs, MSDN Magazine, MSDN Newsletter, TechNet Newsletter)

Load Testing RAG based Generative AI Applications

ogkranthi — Mon, 27 May 2024 18:10:31 GMT

When developing applications for Language Models (LLMs), we usually spend a lot of time on both the development and evaluation phases to ensure the app delivers high-quality responses that are not only accurate but also safe for users. However, a great user experience with an LLM application isn't just about the quality of responses—it's also about how quickly these responses are provided. So, in this discussion, we'll focus on how to evaluate LLM applications for their response times.

The aim of performance evaluation is to proactively test the application to identify and address performance issues before they impact end-users. In the subsequent sections, we will explore performance evaluation in detail. We will discuss building an effective strategy, mastering evaluation techniques, and provide practical guides. Here's what you can expect:

Building an Effective Strategy
Mastering Evaluation Techniques
How-To Guides

Building an Effective Strategy

Each application has unique characteristics, such as user count, transaction volume, and expected response time. Therefore, it's crucial for you to establish an effective evaluation strategy tailored to the specific application you're evaluating.

Before initiating the tests, you need to outline your strategy, which includes determining the aspects to test and the methods to use. This section provides a detailed discussion on these considerations.

Identifying What to Evaluate

Let's start by defining what you are going to test. For example, if the application is fully implemented and running in an environment similar to production, you can conduct a comprehensive load test. This allows you to measure performance and anticipate the user experience before the application is released to end users.

Testing the entire application is a good idea as it provides a measure of response times that closely mirrors what a user will experience when interacting with the application. However, a user's interaction with a Large Language Model (LLM) App involves several elements. These include the application frontend, backend, networking, the LLM model, and other cloud services like databases and AI services.

With that in mind, you have the chance to conduct performance evaluations on particular services even before the whole application is finalized and prepared for deployment. For instance, you can proactively assess the performance of the Large Language Model (LLM) that you plan to incorporate in the application, even if it isn't fully ready yet. In the Mastering Evaluation Techniques section, you will see how you can test the performance of a model deployed in the Azure OpenAI service.

Let's examine an example of an application structure where multiple services cooperate to deliver a user response. One frequently used architectural design in Large Language Model (LLM) Applications like ChatGPT is the Retrieval Augmented Generation (RAG). This architecture involves a retrieval step before invoking the LLM to generate content, which is essential for providing grounding data.

The Enterprise RAG Solution Accelerator architecture offers a practical example of the RAG pattern implemented in an enterprise setting. In the How-To Guides section, you can find an example of load testing on an LLM application that uses the RAG pattern. The following image illustrates the orchestration flow within an LLM application based on RAG. For simplicity, we have not depicted the return messages in the diagram.

Example of communication between the components of an LLM App based on the RAG pattern.

Here's how it works:

The user interacts with the frontend UI to pose a question.
The frontend service forwards the user's question to the Orchestrator.
The Orchestrator retrieves the user's conversation history from the database.
The Orchestrator accesses the AI Search key stored in the Key Vault.
The Orchestrator retrieves relevant documents from the AI Search index.
The Orchestrator uses Azure OpenAI to generate a user response.

Optional connections (dashed arrows):

The connection from the App Service to Storage Account indicates the scenario when the user wants to view the document that grounds the provided answer.
The connection from the App Service to Speech Services indicates the cases when the user wishes to interact with the application through audio.

Each step in the process involves data transfer and processing across various services, all contributing to the total response time. In such scenarios, you can evaluate not just the overall application response time, but also the performance of individual components, like the response times of the Azure OpenAI model deployment.

Ultimately, the scope of testing depends on each application's specific requirements. For instance, an internal application and a public-facing application may have different performance needs. While a response time of 15 seconds might be acceptable in an internal HR application used to view paychecks, a contact center app with hundreds of users might need to respond much faster due to its high user demand and SLAs. Make sure you know the requirements of your application before starting your performance evaluation.

Test Scenario

After determining what you will evaluate in your application, it's essential to define your test scenario. During the LLM App development, we often start our performance tests in a simple way, with a single request. This first test might include measuring the time it takes for completions to appear in the playground, gauging the time a Prompt flow takes to execute in AI Studio, or calculating the time necessary for code execution in VS Code. The data from this single run can help detect potential performance issues in specific components early on.

However, for a genuine perspective on the application's performance, you will need to establish a test scenario that mirrors the actual user load on the application. This entails conducting tests under circumstances that closely resemble real-world usage. By doing so, your assessments will accurately depict how the application would perform under standard operational conditions. We will offer some suggestions on how to accomplish this shortly.

First, we need to determine the load that will be placed on the application. This load is defined in terms of throughput, which is the number of requests the application will receive within a specific time frame, such as Requests per Minute (RPM).

There are multiple ways to estimate the expected throughput. If the application is already operational, you can use its current usage data, gathered from monitoring tools, as a reference. The subsequent figure illustrates this approach. If you foresee an increase in usage due to the integration of LLM into the solution, you should adjust your throughput estimation to accommodate this anticipated growth.

Example of usage scenario, see the peak load from 10h to 13h hours.

When dealing with a new application, estimating the expected throughput can be approached through benchmarking or usage modeling. Benchmarking involves comparing your application with similar ones that serve the same target audience. By studying their usage patterns, you can get a rough estimate of the expected throughput for your application.

Usage modeling, on the other hand, requires you to create a model of the expected usage patterns of your application. This can be achieved by interacting with stakeholders or potential users in the specific field for which the application is being developed. Their insights can provide a better understanding of how the application might be used, which can assist in estimating the Requests Per Minute (RPM).

One approach to model your application usage is starting by identifying the total number of users. This should encompass all registered, or potential users of your application. Then, identify the number of these users who are active during peak usage times. Focusing on peak usage times is crucial as off-peak data may not accurately reflect system performance, particularly for systems with distinct high usage periods.

Next, estimate the average number of times a user will use the application during peak times. This is referred to as sessions. Also, estimate the number of actions or interactions a user makes during a session. Each interaction corresponds to a request made to the application.

For example, consider a mobile app for a shopping mall. Users are likely to have multiple interactions in a single session. They might first search for recommended restaurants, then ask about the operating hours of a specific restaurant. Each of these actions is an interaction.

Example of a user session.

Once you have the total number of users (u), the percentage (p) of them that will use the application during the peak usage hours (n), the average number of user sessions (s), and the typical number of interactions (i) each user has per session, you can use the following formula to derive the RPM to run your load test.

RPM = (u * p * s * i) / n / 60

Taking the previous example of the Mall App, let's consider a set of 10,000 registered users on the App. We expect that during peak hours, 10% of the users will interact at least once with the application to obtain information, such as store locations or product details.

In this case, we have:

u=10000 (total users)
p=0.1 (percentage of active users during peaktime)
s=1 (sessions per user)
i=2 (interactions per session)
n=1 (peaktime duration in hours)

Therefore, the expected throughput for the peak hours is approximately 17 RPM.

Note: During testing, you may want to reproduce a load that is about 10% higher than estimated to be more conservative.

Defining the scenario for larger applications can become complex, requiring the identification of distinct user roles and their usage behavior. However, if exact modeling is challenging due to lack of information, just keep things simple, make an educated guess, and validate it with the application's stakeholders.

Another factor to consider when defining your test scenario is that LLM response times depend on the sizes of prompts and completions. Accurate testing requires replicating real usage scenarios, matching these sizes. For instance, RAG prompts are typically larger due to context text, while proofreading apps usually have similar prompt and completion sizes.

It's worth noting that the examples provided here are primarily based on the perspective of a real-time application user. However, it's also possible to design other performance test scenarios, such as for applications operating in batch mode. In this case, for example, one would need to consider the volume of data to be processed within a specific time window to determine if the application can handle the load within the given time frame.

Whether your application is real-time or batch-oriented, it's crucial to accurately define your test scenario to reflect actual usage conditions. This will ensure that your performance tests yield meaningful results that can help optimize your application's performance.

Test Data

Performance testing heavily relies on the data used during the test execution. It's crucial to use data that closely mirrors real-world scenarios. For instance, if we're testing an application like a copilot, the test results would be more accurate if each user asks different questions. Even if users ask the same questions, they should phrase them differently.

Consider a scenario where each virtual user asks the exact same question during the test. This could lead to results that don't accurately represent real-world usage. Certain components of the application might leverage caching mechanisms to deliver faster responses, skewing the results. Furthermore, the final metric, typically an average or a percentile, will be biased towards the repeated question.

Having experts in the App domain contribute to the creation of the test data set can greatly enhance its quality and relevance. Their knowledge can help shape more realistic and relevant examples. Alternatively, a Large Language Model (LLM) can be utilized to generate a synthetic dataset. This approach can be particularly useful for running tests, as it allows for the creation of diverse and comprehensive data scenarios. This practice not only enhances the quality of the tests but also ensures that they cover a wide range of potential real-world situations.

Test Measurements

Performance testing requires identifying key metrics. A common one is Response Time, the total time from sending a request to receiving a response. Performance requirements are typically determined by this metric, such as needing an average response time under ten seconds, or 95% of responses within ten seconds.

However, response time is not the only metric of interest. To gain a holistic understanding of the application's performance and the factors affecting it, we can categorize the metrics into two groups.

The first group comprises metrics that can be measured from the client's perspective - what the client can observe and capture. The second group consists of metrics that are measured by monitoring the server's performance. Let's explore each of these groups in detail:

Client metrics

When testing an LLM App, we usually obtain the following client metrics:

Metric	Description
Number of Virtual Users	This metric shows the virtual user count during a load test, helping assess application performance under different user loads.
Requests per Second	This is the rate at which requests are sent to the LLM App during the load test. It's a measure of the load your application can handle.
Response Time	This refers to the duration between sending a request and receiving the full response. It does not include any time spent on client-side response processing or rendering.
Latency	The latency of an individual request is the total time from just before sending the request to just after the first response is received.
Number of Failed Requests	This is the count of requests that failed during the load test. It helps identify the reliability of your application under stress.

Note: Response time, often referred to as "end-to-end response time," includes all processing time within the system. However, this measurement only covers the interval between sending a request and receiving the response. It does not account for any client-side processing time, such as rendering a webpage or executing JavaScript in a web application.

Note: Latency and response time are terms that are often used interchangeably, and their definitions can vary depending on the context. In the Azure OpenAI documentation, latency is defined as "the amount of time it takes to get a response back from the model." For consistency, we will use the definition provided in the previous table, which is based on the concepts from Azure Load Testing.

The diagram below illustrates how processing and communication times add up to the total response time. In the figure, each Tn marks a specific time during processing. T1 is when the user initiates a request through a client, such as a browser or MS Teams. T10 is when the user gets the full response. Note that the total response time (from T1 to T10) depends on the processing and response times of all components involved in the request.

Simplified example of the breakdown of request response time.

Performance Metrics for a LLM

When conducting performance testing directly on a specific service, we can collect specific client-side metrics for the target service. In the context of performance testing a Language Model (LLM), we should consider metrics related to prompt tokens and response tokens. For instance, consider the deployment of an OpenAI model on Azure. The following table presents some of these metrics, which offer valuable insights into the client's interaction with the model deployment and its performance under load.

Metric	Description
Number Prompt Tokens per Minute	Rate at which the client sends prompts to the OpenAI model.
Number Generated Tokens per Min	Rate at which the OpenAI model generates response tokens.
Time to First Token (TTFT)	The time interval between the start of the client's request and the arrival of the first response token.
Time Between Tokens (TBT)	Time interval between consecutive response tokens being generated.

Note: To examine the time intervals between tokens in Azure OpenAI's responses, you can utilize its streaming feature. Unlike conventional API calls that deliver the entire response at once, streaming sends each token or a set of tokens to the client as soon as they are produced. This allows for real-time performance monitoring and detailed analysis of the dynamics of response generation.

The diagram below provides a simplified view of a client's interaction with a model endpoint. The interaction commences at the moment (T0) when the client sends a request to the model's endpoint. The model responds in streaming mode, with T1, T2, and TN representing the moments when the first, second, and last tokens are received, respectively.

AOAI deployment response in streaming mode.

In this scenario, we define several key metrics: Time to First Token (TTFT) is T1 - T0, Time Between Tokens (TBT) is T2 - T1, and the end-to-end response time is TN - T0. It's important to note that in streaming mode, the model's responses can arrive in multiple parts, each with several tokens. This makes both the diagram and the metrics an approximate representation of real-world scenarios.

Server metrics

During performance testing, we focus on two types of metrics. The first type is client metrics, which directly affect the user experience. The second type is server metrics, which give us insights into the performance of server components.

Server metrics encompass a wide range of measurements. For instance, we might look at the CPU and memory usage of the application service running the frontend. We could also monitor the utilization of resources like the Azure OpenAI PTU deployment. These are just a few examples; there are many other server metrics we could potentially examine.

By collecting these measurements, we can create a detailed performance profile of the entire solution. This profile helps us identify any bottlenecks and tune any components that are not performing optimally. LLM Apps consist of various services, and the server metrics we utilize will vary based on these services. To give you an idea, here are some examples of the metrics we might gather, depending on the specific service in use:

Service Name	Metric	Description
Azure OpenAI	Azure OpenAI Requests	Total calls to Azure OpenAI API.
Azure OpenAI	Generated Completion Tokens	Output tokens from Azure OpenAI model.
Azure OpenAI	Processed Inference Tokens	The number of input and output tokens that are processed by the Azure OpenAI model.
Azure OpenAI	Provision-managed Utilization V2	The percentage of the provisioned-managed deployment that is currently being used.
Azure App Service	CPU Percentage	The percentage of CPU used by the App backend services.
Azure App Service	Memory Percentage	The percentage of memory used by the App backend services.
Azure Cosmos DB	Total Requests	Number of requests made to Cosmos DB.
Azure Cosmos DB	Provisioned Throughput	The amount of throughput that has been provisioned for a container or database.
Azure Cosmos DB	Normalized RU Consumption	The normalized request unit consumption based on the provisioned throughput.
Azure API Management	Total Requests	Total number of requests made to APIM.
Azure API Management	Capacity	Percentage of resource and network queue usage in APIM instance.

When should I evaluate performance?

You might be wondering when to execute performance tests. To help us in this discussion, let's take a look at the Enterprise LLM Lifecycle, illustrated in the following image.

Enterprise LLM Lifecycle.

The Enterprise LLM Lifecycle with Azure AI involves ideating and exploring, building and augmenting, operationalizing, and managing loops to develop, enhance, deploy, and govern large language model (LLM) applications. You can learn more about the Enterprise LLM Lifecycle by reading this blog: Building for the future: The enterprise generative AI application lifecycle with Azure AI.

Performance testing is crucial and should start as early as possible during the development process. This early start provides enough time for making necessary adjustments and optimizations. The exact timing, however, depends on what aspects of the application you're testing.

If your goal is to evaluate the performance of the entire LLM App before it's used by end-users, the application must be fully developed and deployed to a staging environment. Typically, this load testing of the LLM App occurs during the initial iterations of the Operationalization loop in the Enterprise LLM Lifecycle.

Keep in mind that there are scenarios where performance evaluations can be conducted before Operationalization. For instance, during the Experimenting and Ideating phase, you might be exploring various LLMs for use. If you're considering using one of the models available on Azure OpenAI, this could be an excellent time to conduct a performance benchmark test using the Azure OpenAI benchmarking tool.

The following figure illustrates the moments in the LLM lifecycle where the two types of performance tests mentioned earlier are usually conducted.

Performance tests in the LLM Lifecycle.

Mastering Evaluation Techniques

Great job on your journey so far in learning the essentials of your testing strategy! As we proceed in this section, we will be examining two distinct evaluation techniques. The first technique will concentrate on the performance testing of the entire LLM application, while the second will be primarily focused on testing the deployed LLM. It's important to remember that these are just two popular instances from a wide-ranging list. Depending on your unique performance requirements, integrating other techniques into your testing strategy may prove beneficial.

LLM App Load Testing

Azure Load Testing is a fully managed load-testing service that enables you to generate high-scale LLM App load testing. The service simulates traffic for your applications, regardless of where they're hosted. You can use it to test and optimize application performance, scalability, or capacity of your application. You have the flexibility to create and execute load tests either through the Azure portal or via the Azure Command Line Interface (CLI), managing and running your tests in the way that suits you best.

Azure Load Testing helps you simulate a large number of users sending requests to a server to measure how well an application or service performs under heavy load. You can use Apache JMeter scripts to set up and run these tests. These scripts can act like real users, doing things like interacting with the service, waiting, and using data. In the How-To Guides section, you will find a guide on how you can test your LLM App with a practical example.

The diagram below shows the high-level architecture of Azure Load Testing. It uses JMeter to simulate heavy server loads and provides detailed performance metrics. You can adjust the number of test engine instances to meet your load test requirements, making the system scalable and robust.

Azure Load Testing Overview.

LLM App load testing is crucial for identifying performance issues and ensuring that your application and its Azure dependencies (like the App Service, Function App, and Cosmos DB) can handle peak loads efficiently.

The following table offers an explanation of important concepts associated with Azure Load Testing. Grasping these concepts is essential for effectively using Azure's load testing features to evaluate the performance of the LLM App under various load scenarios.

Concept	Description
Test	Refers to a performance evaluation setup that assesses system behavior under simulated loads by configuring load parameters, test scripts, and target environments.
Test Run	Represents the execution of a Test.
Test Engine	Engine that runs the JMeter test scripts. Adjust load test scale by configuring test engine instances.
Threads	Are parallel threads in JMeter that represent virtual users. They are limited to a maximum of 250.
Virtual Users (VUs)	Simulate concurrent users. Calculated as threads * engine instances.
Ramp-up Time	Is the time required to reach the maximum number of VUs for the load test.
Latency	The latency of an individual request is the total time from just before sending the request to just after the first response is received.
Response Time	This refers to the duration between sending a request and receiving the full response. It does not include any time spent on client-side response processing or rendering.

Azure Load Testing allows for the definition of parameters, which include environment variables, secrets, and certificates. Among its features are test scaling, the setting of failure criteria, and the monitoring of server metrics for application components. Additionally, you can use CSV files to define your test data and upload JMeter configurations for flexible, customizable test scripts.

You can securely store keys and credentials used during the test as Azure Key Vault secrets, and Azure Load Testing can also have its managed identity for access to Azure resources. When deployed within your virtual network, it can generate load directed at your application's private endpoint. Application authentication through access tokens, user credentials, or client certificates is also supported, depending on your application's requirements.

Monitoring Application Resources

With Azure Load Testing, you can monitor your server-side performance during load tests. You can specify which Azure application components to monitor in the test configuration. You can view these server-side metrics both during the test and afterwards on the load testing dashboard. The following figure shows an example of server-side metrics obtained from an App Service after running a test. You can see the Azure services from which you can obtain server-side metrics in this link.

Azure Load Testing Server-side Performance Metrics.

Load Testing Automation

Integrating Azure Load Testing into your CI/CD pipeline is a key step in enhancing your organization's adoption of LLMOps practices. This integration enables automated load testing, ensuring consistent performance checks at crucial points in the development lifecycle. You can trigger Azure Load Testing directly from Azure DevOps Pipelines or GitHub Actions workflows, providing a simplified and efficient approach to performance testing. Below are some examples of commands to automate the creation and execution of a load test.

# Sample command to create a load test az loadtest create \ --name $loadTestResource \ --resource-group $resourceGroup \ --location $location \ --test-file @path-to-your-jmeter-test-file.jmx \ --configuration-file @path-to-your-load-test-config.yaml

# Sample command to run the load test az loadtest run \ --name $loadTestResource \ --resource-group $resourceGroup \ --test-id $testId

For more information on configuring a load test and automating these steps using the Azure Command Line Interface (CLI), refer to the Azure Load Testing CI/CD configuration guide and the Azure CLI reference for load testing.

Key Metrics to Monitor During Load Tests

When conducting load tests, it's crucial to monitor certain key metrics to understand how your application performs under stress. These metrics will help you identify any potential bottlenecks or areas that need optimization. Here are some of the most important ones to keep an eye on:

Request Rate: Monitor the request rate during load testing. Ensure that the LLM application can handle the expected number of requests per second.
Response Time: Analyze response times under different loads. Identify bottlenecks and optimize slow components.
Throughput: Measure the number of successful requests per unit of time. Optimize for higher throughput.
Resource Utilization: Monitor CPU, memory, and disk usage. Ensure efficient resource utilization.

Best Practices for Executing Load Tests

To ensure your load tests are effective and yield meaningful insights, it's worthwhile to review the following recommendations. Here are some key strategies to consider:

Test Scenarios: Create realistic test scenarios that mimic actual user behavior
Ramp-Up Strategy: Gradually increase the load to simulate real-world traffic patterns. The warm-up period typically lasts between 20 to 60 seconds. After the warm-up, the actual load test begins
Think Time: Include think time between requests to simulate user interactions.
Geographical Distribution: Test from different Azure regions to assess global performance.

Performance Tuning Strategies for LLM Apps

This section discusses performance tuning for LLM Apps. Application performance is heavily influenced by design and architecture. Effective structures can manage high loads, while poor ones may struggle. We'll cover various performance tuning aspects, not all of which may be universally applicable.

Application Design

Optimize Application Code: Examine and refine the algorithms and backend systems of your LLM application to increase efficiency. Utilize asynchronous processing methods, such as Python's async/await, to elevate application performance. This method allows data processing without interrupting other tasks.
Batch Processing: Batch LLM requests whenever possible to reduce overhead. Grouping multiple requests for simultaneous processing improves throughput and efficiency by allowing the model to better leverage parallel processing capabilities, thereby optimizing overall performance.
Implement Caching: Use caching for repetitive queries to reduce the application's load and speed up response times. This is especially beneficial in LLM applications where similar questions are frequently asked. Caching answers to common questions minimizes the need to run the model repeatedly for the same inputs, saving both time and computational resources. Some examples of how you can implement this include using Redis as a semantic cache or Azure APIM policies.
Revisit your Retry Logic: LLM model deployments might start to operate at their capacity, which can lead to 429 errors. A well-designed retry mechanism can help maintain application responsiveness. With the OpenAI Python SDK, you can opt for an exponential backoff algorithm. This algorithm gradually increases the wait time between retries, helping to prevent service overload. Additionally, consider the option of falling back on another model deployment. For more information, refer to the load balance item in the Solution Architecture section.

Prompt Design

Generate Less Tokens: To reduce model latency, create concise prompts and limit token output. According to the OpenAI latency optimization guide, cutting 50% of your output tokens can reduce latency by approximately 50%. Utilizing the 'max_tokens' parameter can also expedite response time.

Optimize Your Prompt: If dealing with large amounts of context data, consider prompt compression methods. Approaches like those offered by LLMLingua-2, fine-tuning the model to reduce lengthy prompts, eliminating superfluous RAG responses, and removing extraneous HTML can be efficient. Trimming your prompt by 50% might only yield a latency reduction of 1-5%, but these strategies can lead to more substantial improvements in performance.
Refine Your Prompt: Optimize the prompt text by placing dynamic elements, such as RAG results or historical data, toward the end of your prompt. This enhances compatibility with the KV cache system commonly used by most large language model providers. As a result, fewer input tokens need processing with each request, increasing efficiency.
Use Smaller Models: Whenever possible, pick smaller models because they are faster and more cost-effective. You can improve their responses by using detailed prompts, a few examples, or by fine-tuning.

Solution Architecture

Provisioned Throughput Deployments: When using Azure OpenAI use provisioned throughput in scenarios requiring stable latency and predictable performance, avoiding the 'noisy neighbor' issue in regular standard deployments.
Load Balancing LLM Endpoints: Implement load balancing for LLM deployment endpoints. Distribute the workload dynamically to enhance performance based on endpoint latency. Establish suitable rate limits to prevent resource exhaustion and ensure stable latency.
Resource Scaling: If services show strain under increased load, consider scaling up resources. Azure allows seamless scaling of CPU, RAM, and storage to meet growing demands.
Network Latency: Position Azure resources, like the Azure OpenAI service, near your users geographically to minimize network latency during data transmission to and from the service.

Azure OpenAI Benchmarking

The Improved Azure OpenAI Benchmarking Tool enables you to assess the performance of Azure OpenAI deployments and choose the ideal model and deployment approach (PTU vs. pay-as-you-go) for your specific needs. It simulates various traffic patterns and provides detailed latency metrics. This tool is particularly useful during model selection and experimentation in the initial phases of a project, as it assists developers in determining whether the model deployment is appropriately sized for your project needs.

The Benchmarking tool works by creating traffic patterns that mirror the expected test load. The tool can either automatically generate test requests with random words in the prompt (simulating a workload with a certain number of context tokens), or it can be used with pre-generated messages data, such as data captured or generated from an existing production application. By default, each request is also given a random prefix to ensure that the serving engine processes each request, which is designed to avoid overly positive results that might come from server-side engine optimizations like caching. When conducting the test, it is important to make sure each test runs for long enough for the throughput to reach a stable state, especially when the utilization is close to or at 100%. 180 seconds is generally long enough for most tests.

This kind of load test is particularly useful for deployments that are provisioned managed. By altering the number of units of provisioned throughput (PTUs) that are deployed, you can fine-tune the design of your solution. The results of the analysis may require you to change the number of PTUs, or even the whole structure of the solution. For instance, you may decide to use a combination of PTU and Standard deployments for the Azure OpenAI service, and balance the load between two or more deployments.

Test Parameters

The benchmarking tool contains a number of configuration parameters to configure the test, as well as two script entry points. The benchmark.bench entry point is the basic script point, while the benchmark.contrib.batch_runner entry point can run batches of multiple workload configurations, and will automatically warm up the model endpoint prior to each test workload. It is recommended to use the batch_runner entry point to ensure accurate results and a much simpler testing process, especially when running tests for multiple workload profiles or when testing with PTU model deployments.

The README details the many different options for configuring and running benchmark tests, but some of the key parameters are as follows:

Parameter	Description
rate	Controls the frequency of requests in Requests Per Minute (RPM), allowing for detailed management of test intensity.
clients	Enables you to specify the number of parallel clients that will send requests simultaneously, providing a way to simulate varying levels of user interaction.
context-generation-method	Allows you to select whether to automatically generate the context data for the test (--context-generation-method generate), or whether to use existing messages data for the test (--context-generation-method replay)
shape-profile	Adjusts the request characteristics based on the number of context and generated tokens, enabling precise testing scenarios that reflect different usage patterns. Options include "balanced", "context", "custom" or "generation".
context-tokens (for custom shape-profile)	When context-generation-method = generate and shape-profile = custom, this allows you to specify the number of context tokens in the request.
max-tokens (for custom shape-profile)	This allows you to specify the maximum number of tokens that should be generated in the response.
aggregation-window	Defines the duration, in seconds, for which the data aggregation window spans. Before the test hits the aggregation-window duration, all stats are computed over a flexible window, equivalent to the elapsed time. This ensures accurate RPM/TPM stats even if the test ends early due to hitting the request limit. A value of 60 seconds or more is recommended.
log-save-dir	If provided, the test log will be automatically saved to the directory, making analysing and comparing different benchmarking runs simple.

Warming up PTU endpoints

When testing PTU deployments, it is important to warm up the endpoint prior to the benchmarking workload. This is because PTU endpoints offer a short period of burst capacity until their utilization reaches 100%, after which they will revert back to providing the same throughput as can be calculated with the PTU capacity calculator (accessible in the Quotas tab of Azure OpenAI Studio). To make this process easier, the benchmark.bench.batch_runner entry point will automatically detect and warm-up PTU endpoints, ensuring accurate and realistic results with minimal effort.

Retry Strategy

The retry parameter allows you to set the retry strategy for requests, offering options such as "none" or "exponential", which can be crucial for handling API request failures effectively. When setting up a retry strategy for Azure OpenAI benchmarking, it's crucial to select an approach that carefully balances resource capacity to avoid skewing latency statistics.

When running a test with retry=none, throttled requests are immediately retried with a reset start time, and latency metrics only reflect the final successful attempt, which may not represent the end user's experience. Use this setting for workloads within resource limits without throttling or to assess how many requests need redirecting to a backup during peak loads that surpass the primary resource’s capacity.

Conversely, with retry=exponential, failed or throttled requests are retried with exponential backoff, up to 60 seconds. This approach is only recommended when testing endpoints that have automatic rerouting to a backup resource, and can result in unrealistic latency metrics if used with a test configuration that would ordinarily result in throttling. In general, always use retry=none unless you have backup resources configured behind an endpoint.

Output Metrics

When you run the test, you will obtain average and 95th percentile metrics from the following measures:

measure	description
ttft	Time to First Token. Time in seconds from the beginning of the request until the first token was received.
tbt	Time Between Tokens. Time in seconds between two consecutive generated tokens.
e2e	End to end response time.
context_tpr	Number of context tokens per request.
gen_tpr	Number of generated tokens per request.
util	Azure OpenAI deployment utilization percentage as reported by the service (only for PTU deployments).

Sample Scenarios

**1. Using the benchmark.bench entrypoint**

In the following example, taken from the tool's README, the benchmarking tool tests a traffic pattern that sends requests to the gpt-4 deployment in the 'myaccount' Azure OpenAI resource at a rate of 60 requests per minute, with the retry set to none, and with all logs saved to the logs/ directory. The default traffic shape is used, where each request contains 1000 context tokens, and the maximum response size is limited to 500 tokens.

$ python -m benchmark.bench load \ --deployment gpt-4 \ --rate 60 \ --retry none \ --log-save-dir logs/ \ https://myaccount.openai.azure.com 2023-10-19 18:21:06 INFO using shape profile balanced: context tokens: 500, max tokens: 500 2023-10-19 18:21:06 INFO warming up prompt cache 2023-10-19 18:21:06 INFO starting load... 2023-10-19 18:21:06 rpm: 1.0 requests: 1 failures: 0 throttled: 0 ctx tpm: 501.0 gen tpm: 103.0 ttft avg: 0.736 ttft 95th: n/a tbt avg: 0.088 tbt 95th: n/a e2e avg: 1.845 e2e 95th: n/a util avg: 0.0% util 95th: n/a 2023-10-19 18:21:07 rpm: 5.0 requests: 5 failures: 0 throttled: 0 ctx tpm: 2505.0 gen tpm: 515.0 ttft avg: 0.937 ttft 95th: 1.321 tbt avg: 0.042 tbt 95th: 0.043 e2e avg: 1.223 e2e 95th: 1.658 util avg: 0.8% util 95th: 1.6% 2023-10-19 18:21:08 rpm: 8.0 requests: 8 failures: 0 throttled: 0 ctx tpm: 4008.0 gen tpm: 824.0 ttft avg: 0.913 ttft 95th: 1.304 tbt avg: 0.042 tbt 95th: 0.043 e2e avg: 1.241 e2e 95th: 1.663 util avg: 1.3% util 95th: 2.6%

**2. Using the benchmark.contrib.batch_runner entrypoint**

In the following example, taken from the tool's README, the batch_runner executes the following two traffic patterns for 120 seconds each, making sure to automatically warm up the endpoint prior to each run, and also saving all request input and output content from each run:

context_tokens=500, max_tokens=100, rate=20
context_tokens=3500, max_tokens=300, rate=7.5

With the num-batches and batch-start-interval parameters, it will also run the same batch of tests every hour over the next 4 hours:

$ python -m benchmark.contrib.batch_runner https://myaccount.openai.azure.com/ \ --deployment gpt-4-1106-ptu --context-generation-method generate \ --token-rate-workload-list 500-100-20,3500-300-7.5 --duration 130 \ --aggregation-window 120 --log-save-dir logs/ \ --start-ptum-runs-at-full-utilization true --log-request-content true \ --num-batches 5 --batch-start-interval 3600

For more detailed examples, refer to the README within the repository.

Processing and Analyzing the Log Files

After running the tests, the separate logs can be automatically processed and combined into a single output CSV. This CSV will contain all configuration parameters, aggregate performance metrics, and the timestamps, call status and content of every individual request.

$ python -m benchmark.contrib.combine_logs logs/ combined_logs.csv --load-recursive

With the combined CSV file, the runs can now easily be compared to each other, and with the individual request data, more detailed graphs that plot all request activity over time can be generated.

Monitoring AOAI Resource

Configuring diagnostic settings for Azure OpenAI Service is a good practice for monitoring the availability, performance, and operation of your Azure resources during performance tests. These settings allow the collection and analysis of metrics and log data from your Azure OpenAI resource.

After configuring the diagnostic settings, you can start querying the generated logs. Simply access your Azure OpenAI resource in the portal and then select Logs in the Monitoring section. Next, click on the Log Analytics Workspace that you selected during the diagnostic settings configuration and select the workspace's Logs option.

Below is a query example that retrieves logs from AzureDiagnostics for "ChatCompletions_Create" operations, conducted between 3:30 PM and 4:30 PM on April 26, 2024. It selects logs with details such as timestamp, resource, operation, duration, response code, and additional properties, enabling a detailed analysis of the operation's performance and outcomes during that hour.

AzureDiagnostics | where TimeGenerated between(datetime(2024-04-26T15:30:00) .. datetime(2024-04-26T16:30:00)) | where OperationName == "ChatCompletions_Create" | project TimeGenerated, _ResourceId, Category, OperationName, DurationMs, ResultSignature, properties_s

Analyzing Azure OpenAI Metrics with Azure Monitor.

How-To Guides

Now that you understand the concepts for conducting performance tests, you can refer to the following sections where we provide a detailed guide on how to use the tools mentioned in the text to test your LLM App or your Azure OpenAI model deployment.

LLM RAG application testing with Azure Load Testing.

Model deployment testing with AOAI Benchmarking Tool.

Wrapping Up

In conclusion, performance evaluation is crucial in optimizing LLM applications. By understanding your application's specifics, creating an efficient strategy, and utilizing appropriate tools, you can tackle performance issues effectively. This boosts user experience and ensures that your application can handle real-world demands. Regular performance evaluations using methods such as load testing, benchmarking, and continuous monitoring can lead to your LLM application's ultimate success.

Optimizing ETL Workflows: A Guide to Azure Integration and Authentication with Batch and Storage

Josedobla — Mon, 27 May 2024 17:00:00 GMT

Introduction

When it comes to building a robust foundation for ETL (Extract, Transform, Load) pipelines, the trio of Azure Data Factory or Azure Synapse Analytics, Azure Batch, and Azure Storage is indispensable. These tools enable efficient data movement, transformation, and processing across diverse data sources, thereby helping us achieve our strategic goals.

This document provides a comprehensive guide on how to authenticate Azure Batch with SAMI and Azure Storage with Synapse SAMI. This enables user-driven connectivity to storage, facilitating data extraction. Furthermore, it allows the use of custom activities, such as High-Performance Computing (HPC), to process the extracted data.

The key enabler of these functionalities is the Synapse Pipeline. Serving as the primary orchestrator, the Synapse Pipeline is adept at integrating various Azure resources in a secure manner. Its capabilities can be extended to Azure Data Factory (ADF), providing a broader scope of data management and transformation.

Through this guide, you will gain insights into leveraging these powerful Azure services to optimize your data processing workflows.

Services Overview

During this procedure we will use different services, below you have more details about each of them.

Azure Synapse Analytics / Data Factory

Azure Synapse Analytics is an enterprise analytics service that accelerates time to insight across data warehouses and big data systems. Azure Synapse brings together the best of SQL technologies used in enterprise data warehousing, Spark technologies used for big data, Data Explorer for log and time series analytics, Pipelines for data integration and ETL/ELT, and deep integration with other Azure services such as Power BI, CosmosDB, and AzureML.
Documentation:

Azure Batch

Azure Batch is a powerful platform service designed for running large-scale parallel and high-performance computing (HPC) applications in the cloud.
Documentation: Azure Batch runs large parallel jobs in the cloud - Azure Batch | Microsoft Learn

Azure Storage

Azure Storage provides scalable and secure storage services for various data types, including services like Azure Blob storage, Azure Table storage, and Azure Queue storage.
Documentation: Introduction to Azure Storage - Cloud storage on Azure | Microsoft Learn

Managed Identities

Azure Managed Identities are a feature of Azure Active Directory that automatically manages credentials for applications to use when connecting to resources that support Azure AD authentication. They eliminate the need for developers to manage secrets, credentials, certificates, and keys.
There are two types of managed identities:
- System-assigned: Tied to your application.
- User-assigned: A standalone Azure resource that can be assigned to your app
Documentation: Managed identities for Azure resources - Managed identities for Azure resources | Microsoft Learn

Scenario

Run an ADF / Synapse Pipeline that pulls a script located in a Storage Account and execute it into the Batch nodes using User Assigned Managed Identities (UAMI) for Authentication to Storage and System Assigned Managed Identity (SAMI) to authenticate with Batch.

Prerequisites

ADF / Synapse Workspace

Documentation: Quickstart: create a Synapse workspace - Azure Synapse Analytics | Microsoft Learn

UA Managed Identity

Storage Account

Documentation: Create a storage account - Azure Storage | Microsoft Learn

Procedure Overview

During this procedure we will walk through step by step to complete the following actions:

Create UAMI Credentials
Create Linked Services for Storage and Batch Accounts
Add UAMI and SAMI to Storage and Batch Accounts
Create, Configure and Execute an ADF / Synapse Pipeline

We will refer to ADF (Portal, Workspace, Pipelines, Jobs, Linked Services) as Synapse during all the exercise and examples to avoid redundancy.

Debugging

Procedure

Create UAMI Credentials

1. In your Synapse Portal, go to Manage -> Credentials -> New and fill in the details and click Create.

Create Linked Services Connections for Storage and Batch

2. In your Synapse Portal, go to Manage - Linked Services -> New -> Azure Blob Storage -> Continue and complete the form

a. Authentication Type: UAMI

b. Azure Subscription: Choose your one

c. Storage Account name: Choose your one where the script to be used is allocated

d. Credentials: choose the created into the Step #1

e. Click on Create

3. In Azure Portal go to your Batch Account -> Keys and Copy the Batch Account name & Account Endpoint to be used in next step, also copy the Pool Name to be used for this example.

4. In your Synapse Portal, go to Manage -> Linked Services -> New -> Azure Batch -> Continue and fill in the information

a. Authentication Method: SAMI (Copy the Managed Identity Name to be used later)

b. Account Name, Batch URL and Pool Name: Paste on here the values copied from Step#3

c. Storage linked service Name: Choose the one created from Step#2

5. Publish all your changes

Adding UAMI RBAC Roles to Storage Account

6. In the Azure Portal, go to your Storage Account -> Access Control (IAM)

a. Click on Add Option and then on Add role assignment and search for "Storage Blob Data Contributor", then click on Next.

b. Choose Managed Identity and select your UAMI click on Select and then click Next, Next and Review + assign.

Adding SAMI RBAC Roles to Batch Account

7. In the Azure Portal, go to your Batch Account -> Access Control (IAM)

a. Click on Add Option and then on Add role assignment

b. Click on "Privileged administrator roles" tab and then choose the Contributor role and click Next.

c. Choose Managed Identity and under Managed Identity lookup for "Synapse workspace" and then choose the SAMI same as it is added into the step 4a., then click on Select and Next, Next and Review and Assign.

Adding UAMI to Batch Pool

If you need to create a new Batch Pool, you can follow the following procedure:

Documentation: Configure managed identities in Batch pools - Azure Batch | Microsoft Learn
Make sure to select the UAMI configured into the Step 1

8. If you already have a Batch Pool created follow the next steps:

a. Into the Azure Portal go to your Batch Account -> Pools -> Choose your Pool -> Go to Identity

b. Click on Add then choose the necessary UAMI (on this example it was selected the one used by the Synapse Linked Services for Storage and another one used for other integrations) and click on Add.

Important: In case your Batch Pool use multiples UAMI's (as example to connect with Key Vault or other services), you have first to remove the existing one and then add all of them together.

c. Then, it is required to Scale in and Scale out the Pool to apply the changes.

Setting up the Pipeline

9. In your Synapse Portal, go to Integrate -> Add New Resource -> Pipeline

10. Into the right panel Activities -> Batch Services -> Drag and drop the Custom activities

11. In the Azure Batch tab details for the Custom Activities, click on the Azure Batch linked service and click the one created in Step 4 and test the connection (if you receive a connection error, please go to the Troubleshooting scenario 1)

12. Then go to Settings tab and add your script. Ffor this example, we will use a Powershell script previously uploaded to a Storage Blob Container and send the output to txt file.

a. Command: your script details

b. Resource linked Service: The Storage Service Linked connection configured previously on Step#2

c. Browse Storage: lookup for the Container where your script was uploaded

d. Publish your Changes and perform a Debug

Debugging

12. Check the Synapse Jobs Logs and outputs

a. Copy the Activity Run ID

b. Then, in the Azure Portal Go to your Storage Account -> Containers -> adfjobs -> select the folder with the activityID -> output.

c. On here you will find two files, "stderr.txt" and "stdout.txt" both of them contains information about the errors or the outputs of the commands executed during the task execution

13. Check the Batch Logs and outputs. To get the Batch logs you have different ways:

a. Over Nodes: In Azure Portal go to your Batch Account -> Pools -> Choose your Pool -> Nodes -> then into the Folders details go to the folder for this Synapse execution -> job-x -> lookup for the activityID

b. Over Jobs: In Azure Portal go to your Batch Account -> Jobs -> Select a pool with a name of adfv2-yourPoolName -> click on the Task with the ID same as it was the ActivityID of the Synapse Pipeline from step 12a.

What we have learned

During this walkthrough procedure we have learned and implemented about

Authentication: Utilizing User Assigned Managed Identities (UAMI) and System Assigned Managed Identity (SAMI) for secure connections.
Linked Services: Creation and configuration of linked services for Azure Storage and Azure Batch accounts.
Pipeline Execution: Steps to create, configure, and execute an ADF/Synapse Pipeline, emphasizing the use of Synapse as a unified term to avoid redundancy.
Debugging: Detailed instructions for creating credentials, adding RBAC roles, and setting up pipelines, along with troubleshooting tips.
Logs Analysis: How to access and analyze Synapse Jobs logs and Azure Batch logs for troubleshooting.
Error Handling: Understanding the significance of ‘stderr.txt’ and ‘stdout.txt’ files in identifying and resolving errors during task execution.

If you have any questions or feedback, please leave a comment below!

Microsoft Outlook introduces SMS on Outlook Lite

Keerthana_AK — Mon, 27 May 2024 16:30:00 GMT

Since its launch in 2022, Outlook Lite has provided a way to enjoy the key features of Outlook in a small download size for low-resource phones. We are continuously looking for ways to meet the communication needs of our core users. Now, we are excited to bring SMS on Outlook Lite to users worldwide. With SMS on Outlook Lite, you can enjoy the convenience and security of sending and receiving SMS messages from your Outlook Lite app. SMS is integrated with your email, calendar, and contacts, so you can stay in touch with your contacts in one app.

SMS on Outlook Lite is now available in the latest version of the app, which you can download from the Google Play Store

How to get started with SMS on Outlook Lite?

Getting started with SMS on Outlook Lite is easy and fast. Just follow these steps:

1. Download Outlook Lite from the Google Play Store (here). If you already have Outlook Lite, make sure you update to the latest version.

2. Open Outlook Lite and click on the bottom tab icon named “SMS”

3. Give required permissions to activate SMS.

4. That's it! You can now send and receive SMS messages from Outlook Lite.

What's next for SMS on Outlook Lite?

We are working on adding more features and improvements to SMS on Outlook Lite, such as:

Tighter integration with Email, Calendar and Contacts
Cloud backup of messages
Enhanced Security features.

We would love to hear your feedback and suggestions on SMS on Outlook Lite. You can contact us through the app, or by leaving a comment on this blog post.

Thank you for using Outlook Lite!

Lesson Learned #494: High number of Executions Plans for a Single Query

Jose_Manuel_Jurado — Mon, 27 May 2024 16:29:14 GMT

Today, I worked on a service request where our customer detected a high number of execution plans consuming resources in the plan cache for a single query. I would like to share my lessons learned and experience to prevent this type of issue.

We have the following table definition:

CREATE Table TestTable(ID INT IDENTITY(1,1), string_column NVARCHAR(500)) --We added dummy data in the table running the following script. DECLARE @Total INT = 40000; DECLARE @I int =0 DECLARE @Fill INT; DECLARE @Letter INT; WHILE @i <= @Total BEGIN SET @I=@I+1 SET @Letter = CAST((RAND(CHECKSUM(NEWID())) * (90 - 65 + 1)) + 65 AS INT) set @Fill = CAST((RAND(CHECKSUM(NEWID())) * 500) + 1 AS INT) INSERT INTO TestTable (string_column) values(REPLICATE(CHAR(@Letter),@Fill)) end -- Finally, we created a new index for this column. create index TestTable_Ix1 on TestTable (String_column)

Our customer identified that the application is generating this query:

SELECT TOP 1 * FROM TestTable WHERE string_column = N'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'

To reproduce the issue and understand the impact about how many execution plan our customer reported, we started running the demo function called StartAdhocNoParam: This function executes a non-parameterized query. Running the following DMV to identify the number of plans we could see around 13K cached plans.

-- dbcc freeproccache --Only to clear the cache. WITH XMLNAMESPACES (DEFAULT 'http://schemas.microsoft.com/sqlserver/2004/07/showplan') SELECT qs.sql_handle, qs.execution_count, qs.total_elapsed_time, qs.total_logical_reads, qs.total_logical_writes, qs.total_worker_time, qs.creation_time, qs.last_execution_time, st.text AS sql_text, qp.query_plan FROM sys.dm_exec_query_stats AS qs CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) AS st CROSS APPLY sys.dm_exec_query_plan(qs.plan_handle) AS qp WHERE st.text LIKE '%SELECT TOP 1 * FROM TestTable%'

In this situation, we changed the property of the database called Parameterization to Force, to This resulted in only one execution plan with a parameter. That's is great but our customer wants to modify the source code and avoiding using Parameterization to Force.

Additionally:

OPTIMIZE_FOR_AD_HOC_WORKLOADS might reduce the memory usage, altohough it may not promote the plan reuse - Database scoped optimizing for ad hoc workloads - Microsoft Community Hub

Also, review the option called plan guides that might help on that - Create a New Plan Guide - SQL Server | Microsoft Learn

When our customer finished the modification of their code, we noticed that their application is not specifing the size of parameter or specifing the length of the text that the application is searching, like we could see in the function demo StartAdhocWithParam.

This function is going to run a parametrized query using different length for the parameter, because, for example, if the application is not specifying the length of the parameter or the text that is looking for. In this situation, running the DMV to identify the number of plans we could see around 500 cached plans.

In this situation, we suggested using the function StartParametrize, specifying the max length of the column (500), we could have only an action plan. This reduced the cached plan usage.

This exercise highlights the importance of specifying the length of the parameter,

Finally, I would like to share two new functions:

ImprovedVersionStartParametrize that helps us to reduce the roundtrips of the text sent to the database, only sending values.
GetColumnLength that connects to the database to determine the total size of the column base on the internal table INFORMATION_SCHEMA.columns to perform this more dynamic.

sing System; using System.Data; using Microsoft.Data.SqlClient; class Program { static void Main() { // Parámetros de conexión string connectionString = "Server=tcp:servername.database.windows.net,1433;User Id=username;Password=pwd!;Initial Catalog=dbname;Persist Security Info=False;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30;Pooling=true;Max Pool size=100;Min Pool Size=1;ConnectRetryCount=3;ConnectRetryInterval=10;Application Name=ConnTest"; //ImprovedVersionStartParametrize(connectionString); for (int j = 65; j <= 90; j = j + 1) { Console.WriteLine("Letter:" + (char)j); for (int i = 1; i <= 500; i = i + 1) { if (i % 10 == 0) { Console.Write(" {0} ,", i); } //StartAdhocWithParam(connectionString, (char)j, i); //StartAdhocWithGuide(connectionString, (char)j, i); StartAdhocNoParam(connectionString, (char)j,i); //StartParametrize(connectionString, (char)j, i); } } } static void StartAdhocWithParam(string connectionString, char Letter, int Length) { string query = "SELECT TOP 1 * FROM TestTable WHERE string_column = @stringParam --Adhoc with Param"; using (SqlConnection conn = new SqlConnection(connectionString)) { using (SqlCommand cmd = new SqlCommand(query, conn)) { string stringParam = new string(Letter, Length); cmd.Parameters.Add(new SqlParameter("@stringParam", SqlDbType.NVarChar, Length) { Value = stringParam }); conn.Open(); SqlDataReader reader = cmd.ExecuteReader(); } } } static void StartAdhocNoParam(string connectionString, char Letter, int Length) { string stringParam = new string(Letter, Length); string query = "SELECT TOP 1 * FROM TestTable WHERE string_column = N'" + stringParam + "' --Adhoc without Param"; using (SqlConnection conn = new SqlConnection(connectionString)) { using (SqlCommand cmd = new SqlCommand(query, conn)) { conn.Open(); SqlDataReader reader = cmd.ExecuteReader(); } } } static void StartParametrize(string connectionString, char Letter, int Length) { string query = "SELECT TOP 1 * FROM TestTable WHERE string_column = @stringParam --Adhoc with Max Length"; using (SqlConnection conn = new SqlConnection(connectionString)) { using (SqlCommand cmd = new SqlCommand(query, conn)) { string stringParam = new string(Letter, Length); cmd.Parameters.Add(new SqlParameter("@stringParam", SqlDbType.NVarChar, 500) { Value = stringParam }); conn.Open(); SqlDataReader reader = cmd.ExecuteReader(); } } } static void ImprovedVersionStartParametrize(string connectionString) { string query = "SELECT TOP 1 * FROM TestTable WHERE string_column = @stringParam --Adhoc with Max Length"; using (SqlConnection conn = new SqlConnection(connectionString)) { using (SqlCommand cmd = new SqlCommand(query, conn)) { cmd.Parameters.Add(new SqlParameter("@stringParam", SqlDbType.NVarChar, GetColumnLength(connectionString, "dbo", "TestTable", "string_column"))); conn.Open(); cmd.Prepare(); for (int j = 65; j <= 90; j = j + 1) { Console.WriteLine("Letter:" + (char)j); for (int i = 1; i <= 500; i = i + 1) { if (i % 10 == 0) { Console.Write(" {0} ,", i); } cmd.Parameters[0].Value = new string((char)j, i); SqlDataReader reader = cmd.ExecuteReader(); reader.Close(); } } } } } static void StartAdhocWithGuide(string connectionString, char Letter, int Length) { string query = @" DECLARE @sqlQuery NVARCHAR(MAX) = N'SELECT TOP 1 * FROM TestTable WHERE string_column = @stringColumn'; EXEC sp_executesql @sqlQuery, N'@stringColumn NVARCHAR(500)', @stringColumn = @stringParam"; using (SqlConnection conn = new SqlConnection(connectionString)) { using (SqlCommand cmd = new SqlCommand(query, conn)) { string stringParam = new string(Letter, Length); cmd.Parameters.Add(new SqlParameter("@stringParam", SqlDbType.NVarChar, Length) { Value = stringParam }); conn.Open(); SqlDataReader reader = cmd.ExecuteReader(); } } } static int GetColumnLength(string connectionString, string schemaName, string tableName, string columnName) { using (SqlConnection connection = new SqlConnection(connectionString)) { using (SqlCommand cmd = new SqlCommand(@" SELECT CHARACTER_MAXIMUM_LENGTH FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA = @SchemaName AND TABLE_NAME = @NameT AND COLUMN_NAME = @ColumnName", connection)) { cmd.Parameters.Add("@SchemaName", SqlDbType.NVarChar, 128); cmd.Parameters.Add("@NameT", SqlDbType.NVarChar, 128); cmd.Parameters.Add("@ColumnName", SqlDbType.NVarChar, 128); cmd.Parameters["@SchemaName"].Value=schemaName; cmd.Parameters["@NameT"].Value = tableName; cmd.Parameters["@ColumnName"].Value = columnName; connection.Open(); var result = cmd.ExecuteScalar(); if (result != null) { return Convert.ToInt32(result); } else { return 0; } } } } }

Disclaimer

The use of this application and the provided scripts is intended for educational and informational purposes only. The scripts and methods demonstrated in this guide are provided "as is" without any warranties or guarantees. It is the user's responsibility to ensure the accuracy, reliability, and suitability of these tools for their specific needs.

Remember better with the new Sticky Notes experience from OneNote

chetnadas — Mon, 27 May 2024 16:00:00 GMT

This is a screenshot of new Sticky Notes experience illustrating the new notes creation, 1-click screenshot capture and automatic source capture capabilities

We are excited to announce that the new Sticky Notes experience for Windows is now rolling out to all users. We had first announced the new Sticky Notes experience in this Insiders blog post earlier this year and the response was incredibly positive. Many of you have already started exploring new capabilities of the new Sticky Notes and sharing your feedback, which has been incredibly helpful – thank you.

The new Sticky Notes experience is a fresh feature from OneNote to help you remember more seamlessly than ever. With 1-click screenshot capture, automatic source capture and automatic recall of the notes when you revisit the same source, remembering what matters just got easier! You can also access Sticky Notes on the go with your OneNote Android and iOS mobile apps, ensuring that your notes are always at your fingertips.

How to launch the new Sticky Notes experience

To launch the new Sticky Notes experience, open the ‘OneNote app on Windows’ and click the new Sticky Notes button on top.

This is a screenshot of OneNote ribbon which illustrates the entry point for new Sticky Notes from within the OneNote Windows app

Note: After launching the new Sticky Notes experience, you can pin it to the taskbar. You can also press the Win + Alt + S keys to launch the app anytime.

Soon, you’ll also be able to try the new Sticky Notes experience from the Windows Start menu.

How can new Sticky Notes help you remember better

With the new Sticky Notes, you can create notes or capture screenshots with a single click. If you’ve taken a note or screenshot from a website, you can easily return to the original source by clicking the auto-captured link. When you revisit the same document or website, we'll conveniently bring up the relevant notes for you. Need to multi-task? You can dock the new Sticky Notes to your desktop for a convenient side-by-side experience while using other apps. Search is versatile, including the text within your notes as well as images (using OCR). You can pop out any Sticky Note and view it in a larger window.

For more details, please read the Insiders blog post on new Sticky Notes.

Scenarios to try

At work

When a presentation is shared in a Teams meeting, take screenshots of important slides with a single click, while staying focused on the meeting.
For a recurring meeting, take notes during the meeting and your past notes will automatically surface to the top when you open the new Sticky Notes experience during the next instance of the same meeting series.

When learning

Save important takeaways while watching an educational YouTube video or reading an article. Your previous notes will rise to the top in the app when you return to the same website later.

At home

When planning a trip, take notes and screenshots of potential destinations. The next time you open your notes, click the source link to go back to the website in question for more details or to complete your booking.

Tips and tricks

Pin the new Sticky Notes experience to your taskbar for easy access in the future—no need to launch OneNote.
If you’re already a signed in Sticky Notes user, all your existing notes will appear in the new Sticky Notes experience.
in OneNote app for Windows (click on the profile picture on the top-right) to switch the account associated with your new Sticky Notes .
Sign in to your Microsoft 365 account to sync your notes across your .

Known limitations

“Dock to Desktop” feature does not work well with extended monitor. We’re working to fix this issue soon.

Availability

The new Sticky Notes experience is available to Current Channel users running Windows 10 Version 1903 (SDK 18362) or later, and have OneNote app Version 2402 (Build 17328.20000) or later.

The rollout of this experience is still in progress, and you will get it soon if you haven’t already.

Getting Started with Reliability on Azure: Ensuring Cloud Applications Stay Up and Running

FreddyAyala — Mon, 27 May 2024 13:15:05 GMT

As businesses increasingly rely on cloud services, the imperative for robust cloud solutions has never been greater. Azure stands at the forefront of this realm, offering architects and technology leaders a platform where reliability is not just a feature — it's a core tenet.

The Essence of Reliability in Azure

Reliability is the bedrock upon which cloud architectures stand, indicative of a system's robustness to persistently deliver expected outcomes. It's defined not only by a service’s uptime but also by its stringent adherence to defined Service Level Objectives (SLOs) and Service Level Agreements (SLAs). These crucial benchmarks encompass aspects such as Recovery Time Objective (RTO)—the time within which functions must be restored post-disruption—and Recovery Point Objective (RPO)—the maximum amount of data that can be lost or corrupted post disruption for normal operations to resume. RPO applies not only to storage services but also to other data services such as databases, caches, and queues.

In Azure, reliability means crafting services that are inherently designed to mitigate failures and swiftly rebound from them, with minimal-to-no disruptions experienced by end-users. This is achieved through a shared responsibility model: while Microsoft ensures the underlying infrastructure’s resilience, customers architect their solutions responsibly to exploit these provisions—fusing their understanding of business requirements with Azure's powerful capabilities to uphold service continuity and meet or exceed their RTO and RPO.

The Pillars of Cloud Reliability

The pillars of cloud reliability are critical components of Azure's architecture, designed to ensure dependable service delivery:

Robust Infrastructure: Azure operates a globally distributed network of data centers equipped with advanced redundancy capabilities. This infrastructure is pivotal in providing the resilient physical and virtual resources required for the high availability of applications.
Resilience by Design: Azure's reliability is rooted in its strategic design choices. Solutions architected with resilience in mind are capable of withstanding operational pressures and rapidly recovering from disruptions, ensuring minimal impact on service continuity.
Continuous Operations: Rigorous monitoring, timely incident management, and ongoing system refinement are integral to maintaining the operational health of Azure services. This commitment to continuous operational excellence fortifies service reliability and addresses the evolving demands of cloud workloads.

The Frameworks and Tools Supporting Azure Reliability

Azure's commitment to reliability is underpinned by two foundational frameworks: the Cloud Adoption Framework (CAF) and the Well-Architected Framework (WAF). These frameworks guide organizations through best practices, methodologies, and tools essential for building and maintaining reliable cloud solutions.

Cloud Adoption Framework (CAF):

The CAF provides an extensive set of guidelines, blueprints, and best practices that help streamline the journey to the cloud. It offers insights into readiness and planning, ensuring that foundational decisions support reliability from the outset. Key components include Azure Landing Zones, which configure networking, security, identity, and governance in line with Azure reliability principles.

Well-Architected Framework (WAF):

The WAF focuses on five key areas – cost optimization, operational excellence, performance efficiency, reliability, and security. It empowers architects to design resilient systems by adhering to five principles of architectural excellence in Azure. The reliability pillar of WAF emphasizes the importance of designing systems that are highly available, resilient, and can recover rapidly from failures.

Azure Service Reliability Features:

Each Azure service offers built-in features and tools tailored for enhancing reliability. Some of the important tools are:

Azure Site Recovery: This service ensures business continuity by replicating workloads from primary to secondary regions, facilitating quick failover and minimizing service disruption during outages.
Azure Monitor and Application Insights: Combined, these services provide advanced monitoring, analytics, and diagnostics capabilities, affording real-time operational intelligence that supports swift and proactive incident management.
Azure Automation: With a focus on reducing manual intervention, Azure Automation offers process automation, update management, and configuration features that enhance the reliability of services by eliminating human error.

Architecting for Reliability

Leveraging strategic design choices, Azure enables systems to recover rapidly from disruptions while ensuring continuous operations—a testament to Azure's dedication to non-stop service excellence following our reliability design principles.

Azure Landing Zones: Building Blocks for Reliable Cloud Operations

Azure Landing Zones are pre-defined, customizable environments that follow Microsoft's Cloud Adoption Framework. They provide a structured setup process that incorporates best practices for security, compliance, and governance—forming a reliable foundation for your cloud journey. When setting up your landing zones, consider these reliability-focused factors:

Network Topology: Utilize Azure's robust networking features to design a topology that emphasizes redundancy and failover capabilities.
Resource Organization: Structure your resources for coherence and ease of management, aligning them with your reliability objectives.
Identity and Access Management: Implement tight security controls to prevent unauthorized access which can compromise reliability.
Governance: Establish policies that enforce operational consistency and compliance, adding another layer of reliability protection.

For more information refer to Azure Landing Zones.

Mission-Critical Reliability: Ensuring Resilience at Scale

For mission-critical services where the stakes are especially high, and reliability is imperative, Azure provides a robust toolkit and strategic methodologies to ensure resilience:

Geo-Redundancy: Implementing a multi-region architecture is pivotal for mission-critical applications. Azure facilitates the distribution of services across several geographic locations, safeguarding against regional failures. This approach not only enhances fault tolerance but also enables applications to remain functional and accessible, regardless of localized disruptions.
Disaster Recovery: To safeguard against significant and unexpected disasters, Azure Site Recovery offers a seamless replication service for virtual machines (VMs). It enables swift and structured failovers to alternate regions, ensuring critical applications experience minimal downtime. The service's replication granularity empowers businesses to achieve their specific recovery objectives, be they related to RTO or RPO targets.
Auto-Scaling: Azure's auto-scaling capabilities dynamically adjust resource counts to meet the workload's current demands without human intervention for the services that support this feature. This is essential for meeting performance expectations during usage spikes or unpredicted load increases and for optimizing resource utilization during quieter periods. Such elasticity is vital for maintaining consistent performance levels and operational efficiency.
Monitoring and Diagnostics: Provisioning powerful monitoring tools like Azure Monitor and Azure Application Insights affords organizations real-time visibility into their operational landscape. With these tools, you gain actionable insights, can set up automated alerts for anomaly detection, and pre-empt potential issues based on trends and patterns. The detailed diagnostics provided support rapid issue identification and resolution, which is crucial for mission-critical systems.

By integrating these practices within the architectural fabric, mission-critical services on Azure can achieve the sought-after continuous reliability—delivering consistent service levels and fostering user trust and satisfaction.

For more information refer to Mission Critical Guidance.

Reference Architecture for Reliability

Reliability in the cloud isn't just about having the right tools and services; it's about weaving those elements into an architecture that inherently embodies resilience and fault tolerance. A strategic approach towards crafting a reliable Azure architecture requires a holistic view that spans compute, storage, database, and networking resources.

To illustrate, let's delve into a reference architecture that showcases Azure's reliability principles in action. This architecture demonstrates how various Azure services interconnect to establish a dependable cloud infrastructure, ensuring seamless, continuous operations.

Detailing the Reference Architecture for Reliability

The reference architecture encompasses various Azure services, each contributing to the overall reliability in different ways. Below, we dissect this architecture to understand how the components interrelate and support each other to create a reliable and resilient environment:

Azure Compute Services:

Azure Virtual Machines (VMs): These serve as the backbone, hosting applications and services. To ensure their reliability, leverage Azure Backup, a service offering automated backup solutions that protect VMs from data loss and facilitate easy recovery. Integrating frequent and consistent backups safeguards your data against accidental deletions, corruption, or attacks.
Azure Site Recovery (ASR): Complementing Azure Backup, ASR provides a disaster recovery solution by replicating your Azure VMs to a different availability zone or region. In the event of an outage, you can orchestrate a failover to the replicated VMs situated in the secondary site. This setup ensures minimal downtime and adherence to RTOs (Recovery Time Objectives).

Azure Kubernetes Service (AKS):

Backup and Recovery: The fabric of modern applications often includes containerized solutions orchestrated by AKS. Reliable operation means deploying consistent backups of AKS cluster data, including Persistent Volume (PV) backups, Kubernetes resource configurations, and databases running within the cluster.

Multi-Zone Clusters: AKS supports pod distribution across Availability Zones within a region, ensuring workload continuity in case of a failure in one zone. You can also use services such as Azure Load Balancer or Azure Application Gateway to balance the traffic across zones.
Multi-Regional Clusters: AKS supports deploying clusters across multiple regions, enhancing the resilience and scalability of your applications. You can use services such as Azure Traffic Manager and CosmosDB to distribute the user traffic and data across regions, and orchestrate failover scenarios using Azure Site Recovery.

Azure Storage Services:

Geo-replication: Storage services such as Azure Blob Storage and Azure Queue Storage employ geo-replication strategies to synchronize data across geographically distributed data centers. By doing so, they provide data availability protection against regional outages.
Redundant Storage: Redundancy options, such as Locally-Redundant Storage (LRS) or Zone-Redundant Storage (ZRS), ensure that copies of your data are safely stored within a region or across multiple locations within a region, further fortifying data protection measures.

Azure Database Services:

Automated Backups: Azure services like Azure SQL Database and Azure Cosmos DB offer automated backup features. Automated backups provide a low maintenance approach to protect your databases, enabling the ability to restore databases to a previous point in time quickly in case of data corruption or loss.
Geo-Restore: In addition to regular backups, geo-restore functionalities allow restoration of databases across different geographical regions. In disaster events, this ability is pivotal in maintaining operational continuity and data availability.

By following these architectural principles, you design a robust system that inherently includes resilience and reliability into every layer of its stack. From compute resources down to data storage, the architecture facilitates a cohesive approach to disaster recovery, high availability, and operational effectiveness.

A well-constructed architecture is a critical element in the journey to achieving high reliability on Azure. A reference architecture serves as the blueprint for integrating Azure's resilience principles into your applications. By doing so, you design an ecosystem that not only copes with adverse events but also sustains service continuity and data integrity, thereby meeting high availability standards.

Azure Verified Modules for Reliability

Azure Verified Modules (AVM) is an initiative to consolidate and set the standards for what a good Infrastructure-as-Code module looks like. AVM is a common code base, a toolkit for our Customers, our Partners to accelerate consistent solution development and delivery of cloud-native by codifying Microsoft guidance (WAF), with best practice configurations.

In this article we want to highlight a sample AVM module designed to set up a reliable Azure-to-Azure replication for disaster recovery. It supports replication across regions or within the same region across zone. It’s located at GitHub - Azure/terraform-azurerm-avm-ptn-bcdr-vm-replication: AVM Pattern Module to use Azure Site Recovery to replicate Virtual Machines at Scale between locations. The module replicates virtual machines within Azure from a source to a target location, handling all intermediary replication policies, and resource configurations.

This module provides functionality for:

Creating or using an existing Recovery Services Vault.
Replicating virtual machines between Azure regions or between zones within the same region.
Handling recovery policies, replication policies, and protection container mappings.
Dealing with resource dependencies for orderly creation and deletion.

Conclusion

Commencing your reliability journey on Azure signals a commitment to operational excellence. By leveraging Azure's global infrastructure, proactive design strategies, and a comprehensive suite of tools and best practices, you can pave the way for reliable, scalable, and resilient cloud environments. Elevate your cloud solutions to be ready for any challenge with the power of Azure's reliability features.

Become a champion of reliability and let Azure be the silent force empowering you to deliver steadfast cloud solutions to your stakeholders and customers—today, tomorrow, and into the future.

Thanks to the people that contributed to this article: Harshitha Putta, Laura Grob, Zach Olinske and the PaceSetter reliability team.

Small Language Models with Phi-3 Cookbook: A Guide

Lee Stott — Mon, 27 May 2024 07:00:00 GMT

Small Language Models and Phi-3: A Guide

Small language models (SLMs) are a powerful tool that has gained significant attention in recent years. They are used to generate text, images, and other forms of content with great precision. In this blog post, we will explore the concept of small language models and how you can use them using the Phi-3 cookbook.

What are Small Language Models?

Small language models are a type of neural network that is trained on a large corpus of text data. They are designed to generate new text that is similar to the training data, but they can also be used for tasks like image generation and more.

How do Small Language Models Work?

Small language models work by predicting the next word in a sequence based on the previous words. This process is known as autoregressive prediction. The model learns to generate text by repeatedly predicting the next word in a sequence and comparing its predictions with the actual word.

The Phi-3 Cookbook

The Phi-3 cookbook is an open-source repository that provides a collection of code examples for working with small language models. It includes tutorials, recipes, and other resources that will help you learn how to use these powerful models in your own projects.

What can you Learn from the Phi-3 Cookbook?

The Phi-3 cookbook is a great resource for technical students who want to learn more about small language models. Here are some of the things you can learn from it:

Generating Text with Small Language Models

You can use the Phi-3 cookbook to generate text using small language models. You can learn how to train your own model on a specific dataset, or you can use one of the pre-trained models that is included in the repository. Once you have generated some text, you can use it for tasks like summarization, machine translation, and more.

Generating Images with Small Language Models

You can also use small language models to generate images. The Phi-3 cookbook provides examples of how to use these models to create images from text descriptions. This is a great way to learn about computer vision and image generation.

Applying Small Language Models in Real-World Scenarios

The Phi-3 cookbook includes examples of how small language models can be applied in real-world scenarios. You can learn about how these models are used in applications like chatbots, virtual assistants, and more. This will give you a better understanding of the practical applications of this technology.

Conclusion

Small language models are a powerful tool that is becoming increasingly popular in the tech industry. The Phi-3 cookbook is an excellent resource for technical students who want to learn more about these models and how they can be used in their own projects. By working through the recipes and tutorials in this repository, you will gain valuable knowledge and skills that will help you succeed in your future career.

Customize percentiles for response time in Azure Load Testing

SaloniAgrawal12 — Mon, 27 May 2024 04:42:49 GMT

Introduction

Azure Load Testing is a cloud-based service that lets you test the performance and scalability of your web applications and APIs. You can create and run load tests, analyze and compare the results, and define test fail criteria based on metrics like response time, error percentage, and throughput.

Response time is the time your application takes to respond to a user request. It depends on factors like network latency, server load, and application logic. To measure the performance of your application, you need to look at the response time distribution across different percentiles, which show how many requests are completed within a certain time limit. For example, the 90th percentile response time means that 90% of the requests are completed within that time, and the remaining 10% take longer.

What's new in Azure Load Testing?

Until now, Azure Load Testing only supported three percentiles for response time analysis: Min, Max, Average, Median(50^th), 90th, 95th, and 99th. These percentiles are common in the industry, but they may not match your SLA requirements. For instance, you may want to track the response time at the 75th percentile, which is a standard for web performance, or the 99.9th percentile, which is a stricter measure of the tail latency.

That's why we are happy to announce that Azure Load Testing now supports more percentiles for response time analysis. You can configure the percentiles you want for response time metrics in Azure Load Testing and meet your SLA goals. You can choose from these percentiles:

Min
Max
Average
Median (50th)
75th
90th
95th
96th
97th
98th
99th
99.9th
99.99th

You can use these percentiles in different ways to analyze and compare your test results. For example, you can:

Define test fail criteria based on the percentiles you want for response time metrics.
Filter the ‘Sampler Statistics’ UI by the percentiles you want for response time metrics.
Filter the response time graphs by the percentiles you want for response time metrics.

How to configure percentiles in Azure Load Testing?

Configuring percentiles in Azure Load Testing is easy and intuitive. You can do it in two ways:

Using the Azure portal
Using the YAML file

Using the Azure portal, you can configure the percentiles for response time metrics in the test criteria and the test results pages. In the test criteria page, you can select the aggregate function for the response time metric from the drop-down list that includes all the percentiles as shown below.

In the test results page, you can filter the sampler statistics UI and the response time graphs by the percentiles you want using the filters.

Using the YAML file, you can configure the percentiles for response time when running tests from your CI/CD pipelines. You can find more details on how to configure the YAML in our documentation.

Next Steps

If you haven't tried Azure Load Testing yet, you can create a resource and start testing your application today. If you are already using Azure Load Testing, you can configure the percentiles for your new tests and see the difference. Don't miss this chance to optimize your application performance and deliver a great user experience.

Thank you for choosing Azure Load Testing!

Analyze data using Log Analytics Simple mode

Ilana_Waitser — Mon, 27 May 2024 06:45:05 GMT

Introduction

Azure Monitor Logs offer a powerful set of capabilities for users to explore their logs and derive meaningful insights from their data estate.

Until now, Azure Monitor Logs relied on KQL for users to express their questions as queries.
KQL is a powerful, easy to learn query language, however, as any query language it requires some knowledge to operate.

Simple mode experience was created to bridge this knowledge gap - allowing most popular KQL operators and actions to be utilized using a very simple, point-and-click experience requiring no KQL knowledge at all!

KQL Mode gives advanced users the full power of Kusto Query Language (KQL) to derive deeper insights from their logs.

Here's a video that provides a quick overview of how to query logs in Log Analytics using both Simple and KQL modes:

Try Log Analytics Simple mode

Simple mode is currently an opt-in experience. To try it, select Try the new Log Analytics at the top right corner of the Log Analytics query editor. You can switch back to the classic Log Analytics experience at any time.

Explore and analyze data in Simple mode

Let’s look at the example:

I am an SRE (Site Reliability Engineer), troubleshooting infrastructure issues. For that, I want to understand which Kubernetes pods failed to run.

I just clicked 'Run' on the KubePodInventory table, which brought up the 1000 latest results.

Now, all I need to do is click on Add, under Filter section, search for PodStatus column, select Pending and click Apply.

This brings all pods which have failed to run

Now, I can easily aggregate by Name and see all pod names and how many times they have failed:

I achieved all this without needing to write any KQL code!

Moreover, whenever I select a filter or an operator in Simple Mode, the query runs automatically; there's no need to click on the 'Run' button. This functionality allows for a more fluent experience.

Switch modes

What if you want to make changes to the query and use more advanced operators that are not supported in Simple Mode? No problem!

To do so, we allow to switch from Simple Mode to KQL mode, which allows access to the full power of KQL.

Once I switch to KQL mode, I can see KQL query generated. I can then edit and continue working with the query.

Once I am done with editing, I can switch back to Simple mode and continue the exploration using again the Simple mode on updated query.

Additional Improvements

You will notice some changes aimed at making the UI simple, clean, easy to use, and focused on what matters most – the result set.

One of the changes is organizing the most frequently used actions under separate menus: Save and Share – each of these has sub-actions under it, such as Copy link and Export.

You can find additional actions under '...', such as New Alert or Log Analytics Settings, which enable you to customize behavior according to your needs.

Summary

The new Log Analytics with Simple mode and additional improvements is a huge leap forward in our experiences and we hope you will enjoy using it.

To learn more, we recommend reviewing the feature's official documentation here.

Feedback

We appreciate your feedback!

Please leave comments on this blog post or use the 'Give feedback' in Azure Monitor Logs to share your thoughts with us:

Challenges in Uploading Files Over 2GB via HTTP Protocol in IIS Web Server

Archi_Chakraborty — Sun, 26 May 2024 14:23:20 GMT

Recently, I had worked on a case where customer`s ask was to upload a file larger than 2gb into the IIS hosted web application.

In my case, customer had a normal Asp.net Framework 4.6 application and when they were trying to perform the upload operation of more than 2gb zip file(the extension did not matter...we tested it with .7z and .zip both and it had failed) and somehow on the browser page, we got a Bad request and when I checked the HAR file, i could see a 400 status code.

Next, I checked the application configurations...like:

system.webServer/security/requestFiltering/requestLimits/maxAllowedContentLength system.web/httpRuntime/executionTimeout system.web/httpRuntime/maxRequestLength

but these values were already configured for the higher number.

Articles for references:

HttpRuntime.ExecutionTimeout

MaxAllowedContentLength
Request Filtering

HttpRuntime.maxRequestLength

Also, we checked the App Pool Managed Pipeline Mode, and the interesting part was that we got different error for both the modes.

I was able to recreate this scenario in my lab machine for a sample asp.net framework web application and I had captured FREB logs (Failed request tracing logs) for the site for both the scenarios these were the differences I noticed:

In both the cases, I was able to upload only 2gb file size.
In Integrated Mode, when I tried to upload a 4gb file, I could see this error under the FREB log: "
400 Bad request-Asp.net detected invalid characters in the Url". So, I tried to increase the value of "maxAllowedContentLength" to the maximum supported value, which is 4gb but still it failed, and it seems that webengine code (webengine4!MgdGetRequestBasics) doesn't support more than 2 GB content-length.
Next, in Classic Mode, when I tried to upload a 4gb file, i got this error under FREB log : "413.1 -request entity too large error". So, in this case also, I increased the value of "maxAllowedContentLength" to 4gb, but it failed. We did not see any error under FREB log...it was also 200 status code but the file did not upload.

So, the conclusion is that either you keep the application pipeline mode as Integrated or Classic, you would only be able to upload a 2gb file for your web application hosted on IIS.

If you would like to perform a larger file upload operation, HTTP protocol isn't the right one. You need to switch to webDav feature, or use FTP protocol that is meant to perform the file upload/download operation without any size limit or you can keep using Http protocol but you need to send the data packets as small chunks from client to the server side and then on the server side code, you need to bundle all the chunked packets together for the file upload operation.

Also, note that even if you think moving to latest windows server would fix it...that's not going to help here. This behavior will be same for all the supported IIS across all the versions of supported windows server.

Hope this article helps everyone.

Port 80 Occupied???

Archi_Chakraborty — Sun, 26 May 2024 13:29:31 GMT

Recently I was working on an issue, where customer was not able to browse the Default website hosted on IIS over port 80 and the error was:

Initial thought was, definitely the port is blocked...but who was blocking the port was the biggest question???

So, I started the investigation with basic isolation steps:

Created a different test site with binding port 80 (by stopping the Default website as we cannot run two sites with the same port together). But the test site also failed with the same error.
Created a different IP: Port 80 binding and it worked. Tested it on the default website and there also it worked.
Checked on the customers machine if they had enabled any sort of proxy or firewall that might be blocking the port 80 particularly but that wasn't the case for the customer.
I also ran the Port Query tool and there also I got the error that Port is being used: Port Query Tool
Next, ran the below command to see what all services are listening on port 80 and nothing was useful:

netstat -ano |findstr 80

After doing some research, I found out that we have a service on the windows server called as "Sync Share service " that also by default uses the same port number 80 as IIS Default Web Site.

Now, what is a Sync Share service???

Work Folders is available in Windows Server 2012 R2 Essentials as part of the File and Storage Services role. Work Folders uses the IIS Hostable Web Core feature and all management is performed via the Work Folders canvas in Server Manager as well as via Windows PowerShell cmdlets. Windows Server Essentials is managed via its dashboard and the IIS Management UX. Both products assume exclusive access of the SSL port (443) and HTTP port (80). This is the default configuration for both products.

In my case, the customer had the service enabled but was not using it, so we stopped the service from the Services.msc window and continued using the Port 80 for the IIS hosted web applications.

Now, if you would like to keep using both the services on the server, then you would need to change the port number for one of the services.

We have a very good article written about the same in this article: Port Conflict Issue in 2012 R2

Hope this article helps everyone

Explore Cloud Computing with the New "Azure for Students" Module!

MuhammadSamiullah — Sat, 25 May 2024 07:00:00 GMT

I’m really excited to share a new module that I developed by working with the Microsoft Learn Team: "Introduction to Azure for students." As students, we often encounter numerous tech challenges, from managing projects and assignments to exploring new fields like AI and data science. This module is designed to help you tackle these challenges head-on by diving into the fascinating world of cloud computing with Azure.

Why Cloud Computing?

Cloud computing might sound complex, but it's simpler than you think. Imagine it as a giant library where you can borrow books whenever you need them, without the hassle of buying or storing them. Similarly, cloud computing lets you use computing resources whenever you need them, without owning or maintaining the hardware and software. It's flexible, easy, and often more affordable.

What You'll Learn

In this module, you’ll discover:

Core Concepts of Cloud Computing: Learn the basics and understand what cloud computing is all about.
Azure in Action: Explore real-world scenarios to see how Azure is used in various fields, from student projects to professional healthcare.
Getting Started with Azure: Find out about the tools and services that will help you begin your Azure journey.

Why Azure Cloud?

Azure is Microsoft’s cloud platform, offering over 200 products and services. Whether you're into AI, app development, data science, or machine learning, Azure has it all. Here’s how Azure can help you as a student:

Access Powerful Tools: Utilize advanced tools and services without the need for expensive hardware.
Scalability: Easily scale your resources up or down based on your project needs.
Cost-Effective: Pay only for the resources you use and take advantage of free services and a $100 Azure credit with the Azure for Students offer.

How Azure Can Help with College Projects?

Imagine you’re working on a complex college project that involves data analysis and collaboration with classmates. With Azure, you can set up a virtual environment where everyone can work together seamlessly, share resources, and analyze data in real-time. Need to develop a mobile app for your project? Azure provides all the necessary tools and platforms to build, test, and deploy your app efficiently.

Special Offer for Students!

Being a student usually means living on a budget. Good news—Microsoft's got your back with Azure for Students! Get a $100 Azure credit and access a sea of free services. Learning has never been so cost-effective and fun! Learn more about the Azure for Students offer.

What are you waiting for? Start learning cloud today by going through the module with hands-on experience by claiming the Azure for Students offer. Whether you're working on a school project, developing an app, or just curious about the cloud, this module will provide you with the foundation to succeed.

Ready to get started? Sign up on Microsoft Learn to save your progress and take the first step towards an exciting career in cloud computing!

Build your own copilot! New Microsoft Copilot extensions

Zachary Cavanell — Sat, 25 May 2024 02:40:01 GMT

Customize Copilot for Microsoft 365 with tailored experiences and external data integration through Copilot extensions. Enhance productivity by leveraging large language models and Microsoft Graph for personalized responses. With plugins and connectors, extend Copilot’s capabilities to streamline tasks and automate actions, all while maintaining context and boosting efficiency.

Abram Jackson, Principal Product Manager for Microsoft 365 shows how you can get started building Copilot extensions.

Enrich and focus Microsoft Copilot with Copilot extensions.

How to tailor the user experience with actions, knowledge, and your own copilots.

Automate tasks.

Seamless integration with external systems and tailored experiences. Check out Copilot extensions using connectors and plugins and Microsoft Graph connectors.

Integrate multiple Copilot extensions in one session.

Leverage insights for comprehensive, polished outputs. See how it works.

Watch our video here:

Link References

To get started, check out https://copilotstudio.microsoft.com

Unfamiliar with Microsoft Mechanics?

As Microsoft’s official video series for IT, you can watch and share valuable content and demos of current and upcoming tech from the people who build it at Microsoft.

Subscribe to our YouTube: https://www.youtube.com/c/MicrosoftMechanicsSeries
Talk with other IT Pros, join us on the Microsoft Tech Community: https://techcommunity.microsoft.com/t5/microsoft-mechanics-blog/bg-p/MicrosoftMechanicsBlog
Watch or listen from anywhere, subscribe to our podcast: https://microsoftmechanics.libsyn.com/podcast

Keep getting this insider knowledge, join us on social:

Follow us on Twitter: https://twitter.com/MSFTMechanics
Share knowledge on LinkedIn: https://www.linkedin.com/company/microsoft-mechanics/
Enjoy us on Instagram: https://www.instagram.com/msftmechanics/
Loosen up with us on TikTok: https://www.tiktok.com/@msftmechanics

Video Transcript:

-You can now customize Copilot for Microsoft 365 for your own unique needs, with tailored experiences that can work with external data, while adding unique skills to accelerate the things you do every day. This is now possible with Copilot extensions. And today I’ll demonstrate the experience with Copilot extensions and how to build them.

-First, let me unpack how Copilot for Microsoft 365 works in relation to Copilot extensions. Copilot for Microsoft 365 comprises large language models, information in the Microsoft Graph to find or reference files and data in places like SharePoint, OneDrive, email, your calendar, and Teams, along with information found on the internet to personalize the Copilot experience inside the Microsoft 365 apps you use.

-For example, when you submit a prompt in Word, Copilot determines what you are trying to do, finds additional information to ground your prompt, and then returns a fully formatted, highly personalized response to you. This is what sets Copilot for Microsoft 365 apart from free generative AI experiences.

-Copilot extensions enrich and focus this already great baseline experience further. There are a few main types. The first lets you plug into information sources relevant to you that are sitting outside of Microsoft 365 in near real time, and you can interact with them, like confirming a task was completed.

-This is made possible by available plugins. You can also use connectors to extend what the Microsoft Graph indexes to sites or services also outside of Microsoft 365. The next lets you build your own Copilot, focused on a specific task requiring a specific set of files or data to guide it.

-You can also add additional instructions and workflow to scope what it generates and even automate specific actions. This specialized Copilot is then made available as a Copilot extension for others to use. Let me make this real by walking through how Copilot extensions customize your experience with Copilot for Microsoft 365.

-In this example, I’m a support technician. And one of the things I regularly have to do is write a weekly report detailing the support cases that I’ve worked on. I’m in Copilot for Microsoft Teams, and I’ll first start with my baseline Copilot experience without extensions. I’ll write a prompt to find all the emails I sent today and group them by topic.

-And that finds me all of the emails I was looking for and groups them like I asked. I now want to go beyond Microsoft 365 data and include information and knowledge base articles outside of it that are on our support site. I’ll show you the enriched experience with my Copilot extension first, and then explain what’s going on under the covers.

-I’ll use a different session, and paste in, “Summarize all the emails I sent today, group them by high level topic. For each topic, find the support wiki article that is most related.” And KB articles will come from the connected site. Once the entire response is generated, I can see the matching KB articles for each topic. And if I look at the citations, these ones are the sources using my Graph connected site.

-Here, my Copilot extension is using a Microsoft Graph connector to find information from the external sources that are connected to the Microsoft Graph. Next, I’ll use a different Copilot extension that can also perform actions with an external system on my behalf. My IT team has enabled a plugin called Doc Improvement.

-In this prompt from yesterday, I’ll paste create a doc improvement task to improve that first KB article based on the related emails I sent. This doesn’t need all of the detail from the previous prompt because Copilot gets that information from the chat history in this session. And this extension actually writes back to another system and creates a task for a content writer to update one of our KB support articles.

-After I confirm, I see a confirmation that it has created the doc improvement work item. So far I’ve shown how by using Copilot extensions, my experience as a technician was tailored with content specific to me, while enabling dynamic connections to external data and creating a ticket in my knowledge base system. Now let’s look at how I can go further by focusing in on a specific task with a specialized Copilot available as a Copilot extension.

-This is our support content author Copilot, which my team has built to help us keep our knowledge base articles up to date. You can see that it has a few suggested prompts for skills or frequent tasks. I’ll paste in update the Wiki article about remote help to add a first step of making sure that the user has logged in. This will first find the correct KB article, get its contents, understand where to insert the text, and then ask me to confirm.

-I can also suggest a modification, but let me type, go ahead, and let Copilot make the update. And one more thing I’ll show you is how you can have multiple custom Copilots work together in the same conversation. I’ll go back to the session I had with Copilot earlier. This time, I will at mention the support content author Copilot with a similar prompt.

-This performs the same job, but keeps me in this same session. Now, I’ll at mention another custom Copilot that my team has written called Support Report Writer. Because it is being used in the same session, it has Microsoft Copilot’s analysis of my email, as well as information about the KB article I just updated.

-Then after it’s finished, you’ll see that I now have a fully written draft report that I can copy into an email or a document, and make any additional changes. So, that’s how Copilot extensions work. To build them, you can choose a low-code approach with Microsoft Copilot Studio, or an IDE like Visual Studio. Let’s start with the low-code approach to build your own Copilot. You can get to Copilot Studio by going to copilotstudio.microsoft.com.

-From the Copilots tab, I just need to add a Copilot. And I can either use a prompt to describe the Copilot I want to build, or I can skip right to the configuration, which I’ll do in this case. First, I need to give it a name, so I’ll type in “Support Report Writer.” For description, I’ve enabled clipboard history here, so I’ll paste in the text to describe my Copilot. Next, instructions, give your Copilot more context about its purpose and how to respond. You can get very specific with these instructions using natural language.

-Here I’ve pasted in about 600 characters, but these can be up to 8,000 characters. Starter prompts like we saw before in our custom Copilot can also be added for frequent tasks and to help new users quickly get started. These are optional, and I’ll skip this step to save a little time. From there, I just need to hit create.

-Once complete, I can make it available to the right users and groups by publishing it. Next, you can plug into other data, which can be external to, or inside of Microsoft 365. In the Copilot from Microsoft 365 page, you can see that I already have a few actions, including a few that point to a specific SharePoint locations for content. I’ll add an action.

-Here, I can choose between a connector to link to the data I want, conversation to link to an existing solution, a prompt to link to a standard reusable prompt, and a flow containing workflow automation. I’ll choose a conversation in my case. Now, I’ll give it a name, “Support KB Docs Improvement.” I’ll leave the default solution, and you’ll see that the schema name updates itself automatically.

-Then I’ll hit create. And if you prefer to use an IDE like Visual Studio Code, let me show you the experience with the Microsoft Teams toolkit installed. I’ll start with the plugin this time. You’ll see that I have three basic JSON files open in my tabs. Swagger that JSON to connect to my API using specified operations and responses to send back when it succeeds or fails. Supportplugin.JSON describes the actions to take for users and gives more information for the model to use. Below that, the functions are defined to perform tasks.

-Then in runtimes, it points back to the swagger file and what functions are defined there. Finally, the manifest.JSON, which includes a bit more metadata including the ID, naming, developer, description, and branding details. Importantly, you’ll see that in the manifest, I’ve also defined declarative Copilots. In my case, the content Copilot I showed you earlier. So let’s jump into that JSON file to show you how to build a Copilot.

-You’ll see that this file contains essentially the same elements that we saw in Copilot Studio. Things like the name, ID, and description. The instructions, again, to provide more context for your custom Copilot. We’ve also defined capabilities, which point to our Graph connector data. Actions for a plugin that points to our open API specification, and there are also conversation starters as suggested prompts with details about what those should do.

-And with that, you can get started building Copilot extensions for more tailored experiences, capable of specialized tasks with the data you specify, even if it sits outside of Microsoft 365. These capabilities will be rolling out in the coming weeks. To learn more and get started, check out copilotstudio.microsoft.com. Keep watching Microsoft Mechanics for all the deep dives and latest updates for Copilot and more. Subscribe if you haven’t yet, and we’ll see you soon.

Microsoft Build 2024: Resumo das Principais Novidades em IA e Tecnologia

glaucia_lemos86 — Fri, 24 May 2024 23:51:09 GMT

Durante os dias 21 a 23 de Maio de 2024, aconteceu o evento Microsoft Build 2024, que foi realizado tanto de forma presencial quanto online. O evento contou com inúmeras novidades e incríveis anúncios, principalmente voltados para o uso da Inteligência Artificial em diversas áreas. Neste artigo, estarei abordando os principais destaques do evento, bem como as novidades apresentadas durante o evento.

Então, vamos conferir as principais novidades do Microsoft Build 2024!

Keynote de Abertura - Satya Nadella

O evento já começa destacando a importância e a transformação que a Inteligência Artificial está trazendo para o mundo. Sendo considerado uma das maiores revoluções tecnológicas nos últimos 30 anos.

Logo em seguida Satya Nadella entra no palco para realizar o keynote de abertura do Microsoft Build 2024. Satya Nadella destacou as inovações e lançamentos que prometem transformar a maneira como desenvolvemos e utilizamos a tecnologia com foco em Inteligência Artificial.

Caso queira assistir a keynote de abertura, podem rever abaixo:

Novidades e Anúncios

Copilot+ PCs: O Futuro da Computação com IA

Um dos anúncios mais empolgantes foi o lançamento dos Copilot+ PCs, a nova geração de computadores que integra IA em todos os níveis de hardware e software. Esses dispositivos, estarão disponíveis a partir de 18 de junho, prometem ser até 20 vezes mais poderosos e 100 vezes mais eficientes para cargas de trabalho de IA em comparação com PCs tradicionais.

Em relação a esse lançamento, Satya Nadella apresentou um vídeo demonstrando o poder e a eficiência dos novos Copilot+ PCs. Abaixo vocês podem ver algumas das principais características desses novos dispositivos:

Há uma série de recursos em relação a essa novidade. Para mais informações, acesse o link: Introducing Copilot+ PCs

Windows Copilot Runtime: Integração de IA no Coração do Windows

A Microsoft apresentou o Windows Copilot Runtime, uma plataforma que traz APIs e modelos de IA diretamente para o Windows 11. Com suporte a frameworks populares como PyTorch e a nova estrutura WebNN. Assim sendo, os desenvolvedores poderão criar aplicativos de IA mais eficientes e poderosos.

Para mais informações sobre o Windows Copilot Runtime, acesse o link: Windows Copilot Runtime

Parcerias Poderosas: NVIDIA e AMD

As parcerias da Microsoft com NVIDIA e AMD foram reforçadas, com a introdução de novas GPUs como a NVIDIA H200 e B100, além de VMs baseadas no acelerador AMD MI300X. Essas colaborações visam fornecer a melhor performance e custo-benefício para desenvolvedores que utilizam IA.

Para mais informações sobre essas parcerias, acesse os links: NVIDIA e AMD

Microsoft Fabric: Inteligência em Tempo Real

O Microsoft Fabric foi revelado como uma plataforma integrada para a gestão de dados em tempo real. Com ele, empresas poderão obter insights acionáveis de maneira rápida e eficiente, integrando dados de diversas fontes, incluindo IoT e sistemas de telemetria. Entre as novidades relacionadas ao produto:

Inteligência em Tempo Real: Solução de SaaS para análise de dados em tempo real, permitindo decisões mais rápidas e informadas.
Banco de Dados PostgreSQL no Azure: Capacidades avançadas de IA integradas ao banco de dados, permitindo chamadas para o Azure OpenAI Service e geração de embeddings diretamente no banco de dados.
Novos Recursos no Microsoft Fabric: Kit de Desenvolvimento de Workload, compartilhamento de dados em tempo real, API GraphQL e habilidades de IA integradas para experiências de dados.

Para mais informações sobre as novidades relacionadas ao Microsoft Fabric, acesse o artigo: Unlock real-time insights with AI-powered analytics in Microsoft Fabric

Expansão do Copilot: De Assistente Pessoal a Assistente de Equipe

O Microsoft 365 Copilot foi expandido para atuar como um assistente de equipe, denominado "Team Copilot". Essa ferramenta promete melhorar a colaboração e a produtividade das equipes, facilitando a gestão de reuniões e a comunicação em grupo.

Para mais informações sobre o Team Copilot, acesse o link: Team Copilot

Copilot Studio: Personalização e Automação de Copilots

Com o Copilot Studio, desenvolvedores agora podem criar copilotos personalizados, integrando fontes de dados específicas e automatizando tarefas complexas. A ferramenta também suporta a criação de conectores personalizados, facilitando a integração com sistemas corporativos.

Para mais informações sobre o Copilot Studio, acesse o link: Copilot Studio

Família Phi-3: Modelos de Linguagem Avançados

A Microsoft ampliou sua família de modelos de linguagem, afirmando o seu compromisso em desenvolver soluções Open Source. E, como estamos vivenciando uma demanda crescente por modelos de linguagem, a Microsoft introduzindo o P**hi-3 Vision**, que combina capacidades de linguagem e visão, e novos modelos menores e mais eficientes. Esses modelos prometem ser altamente eficazes em uma variedade de aplicações.

Para mais informações sobre a Família Phi-3, acesse o link do artigo: New models added to the Phi-3 family, available on Microsoft Azure

Windows 365 e Parceria com Meta: Expansão para Novos Dispositivos

A parceria com a Meta permitirá que o Windows 365 seja utilizado em dispositivos Quest, trazendo a experiência do Windows para a realidade virtual e permitindo a criação de aplicativos volumétricos em 3D.

Keynote Dia 2 - Scott Guthrie

O segundo dia do evento foi marcado pela keynote de Scott Guthrie, que destacou a importância da nuvem e da IA especificamente para desenvolvedores. Ele apresentou várias novidades e atualizações relacionadas ao Azure e ao desenvolvimento de aplicativos relacionados à IA.

Caso queira assistir a keynote de Scott Guthrie assista abaixo:

Vamos agora conferir as principais novidades apresentadas por Scott Guthrie

GitHub Copilot Workspace: Revolucionando o Desenvolvimento de Software

Scott Guthrie anunciou o GitHub Copilot Workspace, uma evolução do GitHub Copilot que permite aos desenvolvedores brainstorm, planejamento, construção e teste de código usando linguagem natural. A ferramenta promete aumentar a produtividade e reduzir a complexidade do desenvolvimento de software.

O serviço por enquanto está em fase preview. Mas, você pode saber mais sobre o GitHub Copilot Workspace acessando o link: GitHub Copilot Workspace

AKS Automatic: Kubernetes Simplificado

A nova oferta AKS Automatic tem como proposta automatizar a configuração e gerenciamento de clusters Kubernetes, garantindo desempenho otimizado e segurança. Essa ferramenta é ideal para desenvolvedores que buscam simplicidade e eficiência na gestão de suas aplicações em contêineres.

.NET Aspire: Desenvolvimento em Nuvem Facilitado

A Microsoft lançou o como General Availability o .NET Aspire, uma ferramenta de código aberto que facilita o desenvolvimento de soluções nativas da nuvem, proporcionando segurança, confiabilidade e alto desempenho.

Se você deseja experimentar o .NET Aspire, acesse o tutorial: Quickstart: Build your first .NET Aspire app

E, se você for uma Pessoa Desenvolvedora C# e deseja saber como usar o .NET Aspire com Node.js, acesse o tutorial: Build .NET Aspire apps with Node.js

GitHub Copilot for Azure: Integração Perfeita com a Nuvem

A integração do GitHub Copilot com Azure permite que os desenvolvedores façam perguntas em linguagem natural sobre seus ambientes de nuvem Azure, aumentando a eficiência e mantendo os desenvolvedores no fluxo de trabalho.

Para saber mais sobre o GitHub Copilot for Azure, acesse o link: do artigo: Copilot in Azure Technical Deep Dive

Azure AI Studio: Hub de Desenvolvimento de IA Generativa

O Azure AI Studio é o centro de desenvolvimento para soluções de IA generativa, incluindo o novo serviço de tokens PGO Global, que oferece até 10 milhões de tokens por minuto de throughput, e um desconto de 50% para processamento de lotes fora do horário de pico.

Algumas das novidades apresentadas no Azure AI Studio:

Abordagem de Desenvolvimento Dual: Interface amigável e capacidades de desenvolvimento orientado a código
Experiências de Desenvolvimento: Integração com Azure Developer CLI (azd) e AI Toolkit para Visual Studio Code, facilitando operações de modelos de linguagem (LLMOps). (Em pré-visualização)
Modelos como Serviço (MaaS): Acesso a modelos de fundação de líderes do setor, permitindo a criação de apps de IA generativa sem necessidade de infraestrutura dedicada. (Em pré-visualização)
Ferramentas e Serviços de IA: Integração de dados, orquestração de prompts e avaliação de sistemas, suportando fluxos de trabalho multimodais. (Em pré-visualização)
Rastreamento e Depuração: Melhoria na visão e depuração de fluxos de trabalho de IA. (Em pré-visualização)
Monitoramento de Apps de IA Generativa: Monitoramento de uso de tokens, qualidade e métricas operacionais, com alertas para melhorias contínuas. (Em pré-visualização)

Agora o serviço está como General Availability. Para saber mais sobre o Azure AI Studio, acesse o link: Azure AI Studio

Modelos Frontier e Open Source: Flexibilidade e Escolha

A Microsoft está trazendo uma ampla gama de modelos de IA de código aberto para o Azure AI, incluindo mais de 1600 modelos de parceiros como Hugging Face, Meta, MSR AI e Nvidia, oferecendo flexibilidade e escolha para os desenvolvedores.

Devido a isso, a Microsoft informou que o novo modelo da OpenAI, o GPT-4o, que conta como um omni-modelo capaz de realizar tarefas de visão, texto e falas, estará disponível no Azure AI Studio.

Para saber mais sobre os modelos de IA de código aberto, acesse o link: OpenAI GPT-4o

Azure AI Search e Azure Cosmos DB: Poder para Aplicações RAG

Ferramentas como Azure AI Search e Azure Cosmos DB são essenciais para a criação de aplicações RAG (Retrieval-Augmented Generation), permitindo escalabilidade e alta qualidade nas respostas.

As novidades apresentadas sobre o Azure AI Search:

Capacidade de RAG em Escala: Suporte para tipos de vetor binário e outras funcionalidades de busca vetorial, melhorando a eficiência de armazenamento.
Relevância da Pesquisa: Melhorias na busca vetorial e híbrida, incluindo ponderação de vetores, controle de limiar de pontuação e aumento no tamanho máximo de recuperação de texto, aprimorando a precisão das respostas.
Integrações de Dados e Processamento:
- Vetorização de imagens integrada via Azure AI Vision e novos modelos de embedding no catálogo do Azure AI Studio.
- Conector OneLake para integração direta de dados do Microsoft Fabric ao Azure AI Search, ampliando as fontes de dados que podem ser indexadas e pesquisadas.

As novidades apresentadas sobre o Azure Cosmos DB:

Capacidades de Banco de Dados Vetorial Integradas: Indexação vetorial embutida e busca de similaridade vetorial no Azure Cosmos DB para NoSQL, eliminando a necessidade de um banco de dados vetorial separado. (Disponível em junho, em Preview)
Migração de Conta Serverless para Provisionada: Transição de contas serverless para modo de capacidade provisionada através do portal Azure ou CLI, mantendo acesso total a operações de leitura e escrita. (Em pré-visualização)
Recuperação de Desastres Entre Regiões: Recuperação de desastres no Azure Cosmos DB baseado em vCore para MongoDB, permitindo a criação de réplicas de clusters em outras regiões, com promoção automática em caso de falha. (Em Preview)
Integração com Vercel: Conexão fácil de apps Vercel a bancos de dados Azure Cosmos DB existentes ou criação de novas contas Azure Try Cosmos DB integradas aos projetos Vercel. (Em GA)
SDK Go para Azure Cosmos DB: O SDK Go permite operações em contas Azure Cosmos DB para NoSQL, suportando regiões preferidas, tentativas entre regiões e diagnósticos de solicitações. (GA)

Para saber mais sobre as novidades lançadas sobre Azure Cosmos DB, acesse o link: aqui

Conclusão

O Microsoft Build 2024 foi um evento repleto de novidades e anúncios incríveis, principalmente voltados para a Inteligência Artificial e a nuvem. As parcerias com empresas como NVIDIA e AMD, o lançamento dos Copilot+ PCs e a expansão da família Phi-3 são exemplos de como a Microsoft está investindo em soluções inovadoras e de alto desempenho.

Além disso, a integração do GitHub Copilot com Azure e as atualizações no Azure AI Studio demonstram o compromisso da Microsoft em fornecer ferramentas poderosas e flexíveis para desenvolvedores de todo o mundo.

Claro que só foi possível abordar uma pequena parte das novidades apresentadas no evento. Para saber mais sobre todas as novidades e anúncios apresentados durante o Microsoft Build, existe um book explicando detalhadamente cada uma das novidades. Vocês podem acessar o book através do link: Microsoft Build 2024 - Book of News

Espero que tenham gostado do resumo e que as novidades apresentadas possam inspirar vocês a criar soluções inovadoras e impactantes.

Nos vemos no próximo artigo!

Seamless Identity Integration: Azure API Management with Azure AD B2C (AADB2C)

Andrew_Redman — Fri, 24 May 2024 20:13:33 GMT

Seamless Identity Integration: Azure API Management with Azure (AADB2C)

Introduction

Azure API Management (APIM) is a robust platform for managing and securing your APIs. In this blog post, we will guide you through integrating Azure API Management with Azure Active Directory B2C (AADB2C) for identity management. This integration enhances the security of your APIs by requiring user authentication before access is granted. We will break down the process into three key steps: setting up the Developer Portal to use AADB2C, configuring APIM to use OAuth 2.0 for authorization, and implementing token validation to ensure secure access.

This blog post consolidates information from three separate Microsoft Learn documents, providing a comprehensive, end-to-end guide for setting up AADB2C authentication in your APIM instance. Whether you're accessing your API via cURL, Postman, the Developer Portal, or other clients, this guide will ensure that your APIM instance is configured to handle authentication from AADB2C.

Part One: Setting up Developer Portal with AADB2C

In this section, we will prepare the Developer Portal to use AADB2C for user authentication.

Learning Document

Create Sign-Up and Sign-In Policy in AADB2C
1. Set up a "Sign up and sign in" policy in AADB2C.
  1. For this you can use the same policy for both, just create a "Sign up and sign in" policy in AADB2C
2. Ensure you select all the attributes like Given Name, Surname, and return claims like Email Addresses and User's ObjectID. Here are the minimum requirements:
  1. Collect Attributes: Given Name, Surname
  2. Return claims: Given Name, Surname, Email Addresses, User's ObjectID
Configure Identity in Developer Portal
1. Add AADB2C as an identity provider in the Developer Portal.
  1. Open a new tab and navigate the Azure Portal for APIM and under the Developer Portal, select Identities:
  2. Add a provider by clicking the “Add” button. That will prompt you with a choice of Types, you will want to select “Azure Active Directory B2C”
2. Use the MSAL (Microsoft Authentication Library) client library for authentication, do not worry about the rest of the attributes just yet, we will come back to them.
3. Copy the redirect URL for the Developer Portal.
  1. We will need this to create the backend app registration in the next step.
Create Backend App Registration in AADB2C
1. Register an app to represent your API.
  1. Now, go back to your AADB2C tab, select App Registrations from the list on the left and click “New Registration”.
  2. Enter a name for the app registration, for my example I am calling it `apim-api-backend`
  3. Select the Supported account types that best fit your scenario
  4. Setup the redirect URI (Uniform Resource Identifier) with a Single-page application (SPA) and paste the URI you copied from above.
  5. Click register
2. Create a client secret for this app registration.
  1. Now that the app registration is created, you will need to create a client secret
    1. Open the app registration you just created
    2. On the overview tab, you will see the Application (client) ID, copy that for later.
    3. Then go to Certificates & secrets and create a new client secret. After you hit the Add button, it will show you the 'Value' of your secret. You will need to save this as it will not be shown again for security reasons.
Add Identity Provider in APIM
1. Now head back to APIM tab you left open earlier (or just open a new tab now and navigate to the APIM instance we have been working with), go to Identities, Add Identity Provider page and fill out the client id, and client secret with the values you copied from above
  1. You will also need to fill out the Signin tenant. This is your AADB2C tenant, and it has the following format: <aadb2c-tenant-name>.onmicrosoft.com. After you fill out that text box, it should fill out the Authority for you, but if not, it is <aadb2c-tenant-name>.b2clogin.com.
  2. Next fill out the sign-up and sign-in policy (we created this earlier in Step 1, so go copy the name you used above), if you have the profile editing and password reset policies you can add those here as well.
  3. Click Add
2. Now that you have created everything, you will need to republish the developer portal for these configurations to take effect.
3. It will take a few seconds for the portal to publish, but once it is complete you should be able to navigate to the Developer Portal and see a button to login using AADB2C.
Test configuration
1. Confirm that you can successfully log into the developer portal using an AADB2C login.

Part Two: Configuring APIM for OAuth 2.0 Authorization

Now that we have AADB2C integrated as an Identity into the Developer Portal, let us configure APIM for OAuth 2.0 authorization. This will let us ensure we can send the request with an Authorization header from the developer portal.

Learning Document

Modify Backend App Registration
1. Add scopes that represent the APIs you are exposing.
  1. Go back to the AADB2C tenant and navigate to App Registrations
  2. Find the app you created above (this will be our backend app registration; it represents our Api) and open it up
  3. Go to expose an Api, and then click 'Add a scope'
    1. Create a new scope by giving it a name, and display name and a description
    2. Now click 'Add Scope'
    3. If this is the first scope you are adding it will ask you to create an Application ID URI, you can use the default or put your own in there it will just need to follow URI format
    4. Here is what the ‘Add a scope’ blade looks like
  4. If you have more than one scope you want to add, repeat these steps. The scopes will represent the things the user will be able to do from the API
2. Create a new app registration for the client (APIM Developer Portal).
  1. Create the new app registration and give it a meaningful name, I would suggestion something with 'client' in it
  2. Leave the Redirect uris empty for now and select 'Register'
3. Now that the new app registration is created for the client, you will need to create a client secret like you did for the backend app registration.
  1. Make sure you copy the client id and the secret value so you can use it again when we setup the OAuth server in the next few steps.
Allow Access to Backend App Registration
1. In the client app registration, add permissions to request scopes from the backend app registration.
  1. Open the client app registration, under Api permissions, go to add a Permission
  2. This will give you a list of APIs to select from, use the ‘APIs my organization uses’ filter and find the backend-app you created earlier
  3. Select Delegated Permissions for all the scopes you want to add and select Add permissions
  4. You will then need to grant the admin consent for these new permissions
Add OAuth 2.0 Auth Server in APIM
1. Configure OAuth 2.0 settings in APIM, providing details from app registrations.
  1. Navigate back to the APIM tab and go to the 'Oauth 2.0 + OpenID Connect' section and click the '+ Add' button under the Oauth 2.0 tab
  2. This should open the 'Add OAuth2 service' dialog
    1. Name: This will be how APIM identifies the auth server so use names appropriately
    2. Add a description
    3. The Client registration page URL is the page where users can create and manage their accounts, since AADB2C does not have a direct URL for that (you need to navigate via the azure portal) you can just put `http://localhost`
    4. Next select the Auth grant type, the default is Auth Code, but you will want to make sure to follow what you have set up in the app registrations. In my case I selected both Authorization code and Authorization code + PKCE because I set up my client app registration as a SPA. PKCE stands for Proof Key for Code Exchange
    5. Now enter the authorization endpoint URL, you can get this from the Endpoints page on the Overview tab of an app registration, make sure to select the v2 endpoint
    6. Select POST as the auth request method
    7. Now enter the token endpoint URL, you can also get this from the Endpoints page on the Overview tab of an app registration.
    8. Just use the defaults on the 'Authorization request method’ and ‘Access token sending method’.
    9. In the Default scope, enter the permission that you exposed on the client app registration from the backend app registration just a few steps above this. You must use the fully qualified scope URL as the scope name here
      1. If you have multiple scopes, you can list them all here…space separated
      2. It will have this format <Application ID URI>/<permission name>
    10. Next, fill out the clientId and clientSecret from the client app registration that you saved a moment ago
    11. This will generate the Redirect URI
      1. **Important Step** You will need to go back to the client app registration and add this Redirect URI that will need to be added back to the client app registration
    12. Hit Create and you are done
2. Now you should have a new OAuth 2.0 server, you will need to republish the developer portal at this point to make sure the latest configuration is deployed
Add OAuth 2.0 server to your API
1. Go to APIM and navigate to APIs -> Setting tab and then scroll down to the Security section. Select User authorization to Oauth 2.0 and then in the dropdown list select the Oauth server that we created above.
  1. If you need to use different scopes for a given Api you can override it here at this level using the Override scope checkbox
Test in Developer Portal

Log in using AADB2C credentials in the Developer Portal.
Access APIs and see OAuth 2.0 authorization in action.
1. In the Try It section you will see something like this:
  2. Hit the dropdown and select the grant type you setup (in our case it is authorization code)
  4. That should trigger an auth call which will pull the token from your initial login (or prompt you for a new login) and transform it appropriately and populate the 'Authorization' header in the test request.

Part Three: Adding Token Validation for Security

The last step ensures secure API access by adding token validation by adding a validation policy.

Learning Document

Add Validate-JWT Policy
1. In APIM, add a validate-jwt policy to the inbound policy.
  1. Go to the Api you want to add the validation to, or you could set it at the global level:
  2. Use the `validate-jwt` policy. You can find the documentation for it here. This is screenshot of what it might look like, I am using `Named Values` for some variables.
2. Update the <openid-config URL> property with the correct AADB2C endpoint.
  1. After you add that policy, you will want to update the <openid-config URL> property. The format looks like this:
    1. ```
    https://<tenant-name>.b2clogin.com/<tenant-name>.onmicrosoft.com/<user-flow-name>/v2.0/.well-known/openid-configuration
```
2. If you want to copy and paste it, you can get this endpoint from the Endpoints section on the overview tab of the app registration. It is under the `Azure AD B2C OpenID Connect metadata document`. You will just need to fill out the policy name for you signin policy you created above.
3. Configure Audience and Issuers
  1. Specify the audience and issuers for token validation.
4. Testing and Security:
  1. If the JWT (JSON Web Token) is valid, APIM allows the request to continue to the backend.
  2. If the JWT is invalid, APIM blocks the request, enhancing security.

Conclusion

By following these steps, you can seamlessly integrate Azure API Management with Azure ADB2C, ensuring that your APIs are secure and accessible only to authorized users. This setup offers a robust authentication and authorization mechanism for your APIs, making them more reliable and trustworthy.

Updates to Security Admin permissions for Microsoft Copilot for Security

HarmonyMabrey — Fri, 24 May 2024 19:37:26 GMT

Late June 2024, Security Admins will have an expanded authority to control Microsoft security products to access Microsoft 365 Customer Data. Within Microsoft Copilot for Security, the purpose of this is to allow users to query information directly from those M365 products in the standalone and embedded experiences.

Security Admins can choose not to allow Copilot for Security access to your Microsoft 365 services in the owner settings at any time. When access is disabled, users won't be able to use specific plugins, like Microsoft Purview, or connect to any future integrated or embedded Microsoft 365 services.

This change will take effect June 24th. Other admin roles will not be affected by this change.

Copilot for Microsoft 365 Integration with Service Now – HLS Copilot Snacks

Mike Gannotti — Fri, 24 May 2024 19:01:35 GMT

The ServiceNow plugin for Copilot for Microsoft 365 allows users to easily create and track tickets for IT support within the Copilot app. Users can access the plugin from the Copilot toolbar and submit requests with relevant information, such as screenshots, device details, and error messages. The plugin also enables users to view the status of their tickets, communicate with IT agents, and provide feedback on the resolution. This integration can benefit users by simplifying the process of requesting and receiving IT support and can benefit IT support by reducing the need for manual data collection and improving customer satisfaction.

In this Copilot Snack I demonstrate the use of the Copilot/ServiceNow integration from within Microsoft Teams, Microsoft Word, and Microsoft PowerPoint.

*Special thanks to Microsoft's Maria Kurian for enabling the integration that made this demo possible!

To see all HLS Copilot Snacks video click here.

Resources:

Connect Copilot for Service to ServiceNow - Copilot for Service | Microsoft Learn
Combining GenAI Capabilities with Microsoft – ServiceNow Press
Plugins for Microsoft 365 Copilot (Inside Microsoft Teams) – Microsoft Adoption
Prompts:
- Why can't I get my weather report? Are there any tickets in ServiceNow related to weather?
- List the last 5 tickets in ServiceNow and provide a brief description of each
- last 7 tickets in ServiceNow and describes each on a slide

To see all HLS Copilot Snacks video click here.

Thanks for visiting – Michael Gannotti LinkedIn

Lesson Learned #493: Monitoring Application Performance with Server Performance Counters

Jose_Manuel_Jurado — Fri, 24 May 2024 18:59:43 GMT

Today, I worked on a service request where our customer reported several performance issues in their application connecting to Azure SQL Database. After an in-depth analysis, we found that the issue could be related to the server running the application, including resources assigned, network issues, etc.,. Aside from other tools offered by Azure, following, I would like to share the lessons learned using logman that is part of performance monitor tool (perfmon).

Tools Overview

perfmon

The Performance Monitor (perfmon) is a Windows tool that provides a visual interface for monitoring system performance. It can track various performance metrics like CPU usage, memory usage, disk activity, and network activity in real-time.

logman

logman is a command-line tool in Windows that allows you to create and manage performance data collection sets. It can be used to automate the collection of performance data, making it an excellent tool for scheduled and long-term monitoring.

Step-by-Step Guide

Step 1: Create a Data Collector Set with logman

The following script shows how to create a data collector set using logman, configure it to collect various performance counters, and store the data in a CSV file.

Save this script as a Windows Command Batch file (.cmd) and execute it as Administrator:

logman create counter MyDataCollector -f csv -o "C:\Counters\MyDataCollector" logman update MyDataCollector -c "\Memory\Available MBytes" logman update MyDataCollector -c "\TCP(*)\*" logman update MyDataCollector -c "\Network Interface(*)\*" logman update MyDataCollector -c "\Process(*)\*" logman update MyDataCollector -c "\Process(_Total)\*" logman update MyDataCollector -c "\Processor Information(*)\*" logman update MyDataCollector -c "\Processor(_Total)\*" logman update MyDataCollector -c "\Processor(0)\*" logman update MyDataCollector -si 00:00:05 logman start MyDataCollector timeout /t 30 logman stop MyDataCollector logman delete MyDataCollector

Once the application finished, I checked the CSV file generated and found useful information saved in this file, such as network activity, process activity, and resource usage. In this case, we have generated the file C:\Counters\MyDataCollector_000001.csv

Step 2: Import Data into Power BI

Once the data is collected and saved in a CSV file, you can import it into Power BI for analysis.

Open Power BI Desktop.
Get Data:
- Click on "Get Data" and select "Text/CSV".
- Browse to C:\Counters\MyDataCollector_000001.csv and import the file.
Transform Data:
- Power BI will load a preview of the data. Click on "Transform Data" to clean and format the data if necessary.
- You might want to rename columns, change data types, or filter out unnecessary data.
Create Visualizations:
- Once the data is loaded, you can start creating visualizations. Use charts, graphs, and tables to analyze the performance data.
- For example, you can create line charts to visualize memory usage over time, bar charts for network traffic, and pie charts for process usage.

Disclaimer

Microsoft Security Exposure Management Graph: unveiling the power

andreykarpovsky — Fri, 24 May 2024 18:28:03 GMT

Introduction

In the complicated and rapidly evolving realm of cybersecurity, Exposure Management plays a pivotal role in fortifying organization's defenses against potential threats. To empower security teams, Microsoft Security Exposure Management has unveiled two new powerful tables within Advanced Hunting: ExposureGraphNodes and ExposureGraphEdges.

The introduction of these tables opens novel capabilities for security teams. It enables efficient investigation of security posture across organizational assets. This is the first in a series of posts where we will present the tables and share investigation scenarios (along with relevant queries) for Advanced Hunting. These queries unlock capabilities that were previously unattainable. We’ll provide screenshots and Kusto Query Language snippets to guide you through your reading.

Understanding the tables

As John Lambert's saying that is well-known in the security domain goes, 'Defenders think in lists. Attackers think in graphs. As long as this is true, attackers win.’ By exposing the context around each asset, relations between assets and the graph-based toolset for exploring them, we hope to start changing this.

Context: Beyond Asset Information

Think of assets and entities as points on a graph, and of relations between them as links between these points. Traditionally, when dealing with an asset (such as a server, device, or network component), we have focused on gathering specific information about that asset. With the powerful combination of the Attack Map and Extended Security Posture Management tables, we can gain a better perspective that zooms out and reveals the entire environment surrounding each asset.

The Tables:

ExposureGraphNodes

This table represents all the nodes in the Attack Surface Map. According to the table documentation, ExposureGraphNodes contain organizational entities and their properties. These entities include devices, identities, user groups, and cloud assets (such as virtual machines, storage, and containers). Each node corresponds to an individual entity and encapsulates information about its characteristics, attributes, and security-related insights within the organizational structure.

Before diving into specific scenarios, we recommend examining the available data first.

Running the following query will retrieve one node for each type in the environment. This will allow you to see the kinds of nodes present and their properties.

ExposureGraphNodes | summarize take_any(*) by NodeLabel

ExposureGraphEdges

This table represents all the edges in the Attack Surface Map. Each edge describes a pairwise relationship between two of the ExposureGraphNodes we have just reviewed. As stated in Advanced Hunting documentation: “The ExposureGraphEdges schema, along with the complementing ExposureGraphNodes schema, provides visibility into relationships between entities and assets in the graph. Many hunting scenarios require exploration of entity relationships and attack paths. For example, when hunting for devices exposed to a specific critical vulnerability, knowing the relationship between entities, can uncover critical organizational assets.”

Similarly, we recommend exploring your data related to edges. Running the following query will retrieve one edge for each edge type in your organization. After running the query, you’ll gain insights into the relations between your organizational entities and the additional data on them.

ExposureGraphEdges | summarize take_any(*) by EdgeLabel

Now we would like to describe several security-related scenarios that can be investigated using the security exposure graph.

Scenarios

Scenario 1: Nodes with specific properties

Security Exposure Graph displays various assets and entities in your organization (such as storage accounts, devices and users) as nodes in ExposureGraphNodes table, while various properties (such as criticality, sensitive data) appear as node properties.

For proper posture management, it might be interesting to find all nodes filtered by specific types and/or properties. For example, we might want to find all critical assets, or all virtual machines that are exposed to the internet and have vulnerabilities.

Example 1A: Critical assets

Query:

ExposureGraphNodes | project NodeLabel, NodeName, NodeId, Categories , criticalityLevel = toint(NodeProperties.rawData.criticalityLevel.criticalityLevel) | where criticalityLevel > 0 | sort by criticalityLevel desc

Output:

Example 1B: Virtual Machines with specific vulnerabilities (RCE and privilege escalation)

Query:

ExposureGraphNodes | where NodeLabel == 'microsoft.compute/virtualmachines' | project NodeLabel, NodeName, NodeId, NodeProperties , vulnerableToRCE = isnotnull(NodeProperties.rawData.vulnerableToRCE.type) , vulnerableToPrivilegeEscalation = isnotnull(NodeProperties.rawData.highRiskVulnerabilityInsights.vulnerableToPrivilegeEscalation) | where vulnerableToRCE > 0 or vulnerableToPrivilegeEscalation > 0

Output:

Since filtering the assets by specific types and properties can cover various scenarios, it might be useful to wrap such queries in a generic format, that will allow repeated usage with various parameters:

let XGraph_NodesWithTypesAndProperties = (nodeTypes:dynamic, nodeProperties:dynamic) { let propertiesFormatted = strcat('(', strcat_array(nodeProperties, '|'), ')'); ExposureGraphNodes | where NodeLabel in (nodeTypes) or nodeTypes == "[\"\"]" | project NodeName, NodeLabel, NodeId, Categories , propertiesExtracted = iff(nodeProperties != "[\"\"]", extract_all(propertiesFormatted, tostring(NodeProperties)), pack_array('')) | mv-apply propertiesExtracted on ( summarize propertiesExtracted = make_set_if(propertiesExtracted, isnotempty(propertiesExtracted)) ) | extend countProperties = coalesce(array_length(propertiesExtracted), 0) | where countProperties > 0 or nodeProperties == "[\"\"]" | sort by countProperties desc };

Sample usage – find all vulnerable Virtual Machines:

XGraph_NodesWithTypesAndProperties( nodeTypes=pack_array('microsoft.compute/virtualmachines') , nodeProperties=pack_array('vulnerableToRCE', 'vulnerableToPrivilegeEscalation'))

Note that if any of the parameters is an empty array, the function will not filter on it and bring assets of all types or properties. For example, the following will bring vulnerable assets of any type that have RCE or Privilege Escalation vulnerabilities:

XGraph_NodesWithTypesAndProperties( nodeTypes=pack_array('') , nodeProperties=pack_array('vulnerableToRCE', 'vulnerableToPrivilegeEscalation'))

Alternatively, you can create and save several specific functions for common usage and use them without any additional parameters.

let XGraph_VulnerableVMs = () { let nodeTypesList = pack_array('microsoft.compute/virtualmachines'); let nodePropertiesList = pack_array('vulnerableToRCE', 'vulnerableToPrivilegeEscalation'); XGraph_NodesWithTypesAndProperties(nodeTypes = nodeTypesList, nodeProperties = nodePropertiesList) };

Usage -

XGraph_FindVulnerableVMs()

Any function (with or without parameters) can be saved for repeated usage as described here: Custom functions in the advanced hunting schema - Microsoft Defender XDR | Microsoft Learn.

Scenario 2: Connected nodes with specific properties

The relations between assets appear as edges in ExposureGraphEdges table. For example, users that can access a virtual machine will be connected to it with ‘has permissions to’ edge.

We might want to look for pairs of connected assets while filtering by relevant edge types. For example, we might want to find users that have permissions to access keyvaults.

Example 2A: Users that have access to keyvaults

Query:

ExposureGraphEdges | where EdgeLabel == 'has permissions to' and SourceNodeLabel == 'user' and TargetNodeLabel == 'microsoft.keyvault/vaults' | project SourceNodeName, SourceNodeLabel, SourceNodeId, EdgeLabel, TargetNodeName, TargetNodeLabel, TargetNodeId

Output:

Alternatively, we might want to filter not only by endpoint types, but also by properties. For this, we need to join the edges table with node table (that contains the node properties) both on source and target. Note that the unique identifier of each asset is the NodeId (and not NodeName).

Example 2B: Critical users that can access storage accounts with sensitive data

Query:

ExposureGraphEdges | where EdgeLabel == 'has permissions to' and SourceNodeLabel == 'user' and TargetNodeLabel == 'microsoft.storage/storageaccounts' | project SourceNodeName, SourceNodeLabel, SourceNodeId, EdgeLabel, TargetNodeName, TargetNodeLabel, TargetNodeId | join kind = leftouter (ExposureGraphNodes | project SourceNodeId = NodeId, SourceNodeProperties = NodeProperties) on SourceNodeId | join kind = leftouter (ExposureGraphNodes | project TargetNodeId = NodeId, TargetNodeProperties = NodeProperties) on TargetNodeId | extend sourceCriticalityLevel = toint(SourceNodeProperties.rawData.criticalityLevel.criticalityLevel) , targetSensitiveData = isnotempty(TargetNodeProperties.rawData.containsSensitiveData.type) | where sourceCriticalityLevel > 0 and targetSensitiveData > 0 | project SourceNodeName, SourceNodeLabel, SourceNodeId, EdgeLabel, TargetNodeName, TargetNodeLabel, TargetNodeId, sourceCriticalityLevel, targetSensitiveData

Output:

We can add (and save) a generic function that looks for edges between nodes with specific types and properties as well.

let XGraph_EdgesWithTypesAndProperties = (sourceTypes:dynamic, sourceProperties:dynamic, targetTypes:dynamic, targetProperties:dynamic) { let sourcePropertiesFormatted = strcat('(', strcat_array(sourceProperties, '|'), ')'); let targetPropertiesFormatted = strcat('(', strcat_array(targetProperties, '|'), ')'); let edgeTypes = pack_array('has role on', 'has permissions to', 'can authenticate to', 'can authenticate as' , 'member of', 'contains'); ExposureGraphEdges | where EdgeLabel in (edgeTypes) | where (SourceNodeLabel in (sourceTypes) or sourceTypes == "[\"\"]") and (TargetNodeLabel in (targetTypes) or targetTypes == "[\"\"]") | project SourceNodeName, SourceNodeLabel, SourceNodeId, EdgeLabel, TargetNodeName, TargetNodeLabel, TargetNodeId | join hint.strategy = shuffle kind = leftouter (ExposureGraphNodes | project SourceNodeId = NodeId, SourceNodeProperties = NodeProperties) on SourceNodeId | join hint.strategy = shuffle kind = leftouter (ExposureGraphNodes | project TargetNodeId = NodeId, TargetNodeProperties = NodeProperties) on TargetNodeId | extend sourcePropertiesExtracted = iff(sourceProperties != "[\"\"]", extract_all(sourcePropertiesFormatted, tostring(SourceNodeProperties)), pack_array('')) , targetPropertiesExtracted = iff(targetProperties != "[\"\"]", extract_all(targetPropertiesFormatted, tostring(TargetNodeProperties)), pack_array('')) | mv-apply sourcePropertiesExtracted, targetPropertiesExtracted on ( summarize sourcePropertiesExtracted = make_set_if(sourcePropertiesExtracted, isnotempty(sourcePropertiesExtracted)) , targetPropertiesExtracted = make_set_if(targetPropertiesExtracted, isnotempty(targetPropertiesExtracted)) ) | extend countSourceProperties = coalesce(array_length(sourcePropertiesExtracted), 0) , countTargetProperties = coalesce(array_length(targetPropertiesExtracted), 0) | where (countSourceProperties > 0 or sourceProperties == "[\"\"]") and (countTargetProperties > 0 or targetProperties == "[\"\"]") | project SourceNodeName, SourceNodeLabel, SourceNodeId, EdgeLabel, TargetNodeName, TargetNodeLabel, TargetNodeId , sourcePropertiesExtracted, countSourceProperties, targetPropertiesExtracted, countTargetProperties | sort by countSourceProperties desc, countTargetProperties desc };

Sample usage – critical users that have access to containers or storage accounts that are either critical or have sensitive data:

XGraph_EdgesWithTypesAndProperties( sourceTypes = pack_array('user') , sourceProperties = pack_array('criticalityLevel') , targetTypes = pack_array('container', 'microsoft.storage/storageaccounts') , targetProperties = pack_array('containsSensitiveData', 'criticalityLevel'))

This function can also be wrapped and used as is for common scenarios. For example, we can create the following function to cover the scenario above:

let XGraph_CriticalUsersToCriticalOrSensitiveStorage = () { let sourceTypesList = pack_array('user'); let sourcePropertiesList = pack_array('criticalityLevel'); let targetTypesList = pack_array('container', 'microsoft.storage/storageaccounts'); let targetPropertiesList = pack_array('containsSensitiveData', 'criticalityLevel'); XGraph_EdgesWithTypesAndProperties(sourceTypes = sourceTypesList, sourceProperties = sourcePropertiesList , targetTypes = targetTypesList, targetProperties = targetPropertiesList) };

Usage -

FindCriticalUsersToCriticalOrSensitiveStorage()

Scenario 3: Paths between nodes with specific properties

Sometimes nodes can be connected in a non-direct way. For example, a virtual machine can have access to a keyvault using SSH key or managed identity. Alternatively, user can have permissions to a subscription containing storage accounts – thus gaining access to all of them.

A great way to explore such connections and find the multi-step paths is using Kusto graph capabilities – namely the make-graph and graph-match operators (you can learn more about Kusto graph semantics here). These operators allow to build paths between endpoints (source and target nodes) according to conditions on endpoints or any of the steps.

Example 3A: Users that have access to storage accounts with sensitive data

Query:

ExposureGraphEdges | where EdgeLabel in ('has role on', 'has permissions to', 'can authenticate to', 'can authenticate as', 'member of', 'contains') | make-graph SourceNodeId --> TargetNodeId with (ExposureGraphNodes | project NodeId, NodeName, NodeLabel, NodeProperties) on NodeId // Look for existing paths between source nodes and target nodes with less than predefined number of hops | graph-match (s)-[e*1..4]->(t) where (s.NodeLabel == 'user' and t.NodeLabel == 'microsoft.storage/storageaccounts' and isnotnull(t.NodeProperties.rawData.containsSensitiveData.type)) project SourceName = s.NodeName , SourceType = s.NodeLabel , SourceId = s.NodeId , SourceExposedToInternet = s.NodeProperties.rawData.exposedToInternet.type , TargetName = t.NodeName , TargetType = t.NodeLabel , TargetId = t.NodeId , TargetcontainsSensitiveData = t.NodeProperties.rawData.containsSensitiveData.type , edgeIds = e.EdgeId , edgeLabels = e.EdgeLabel | extend pathLength = array_length(edgeIds) + 1

Output:

Example 3B: SQL servers or managed instances with basic authentication that have access to keyvaults

Query:

ExposureGraphEdges | where EdgeLabel in ('has role on', 'has permissions to', 'can authenticate to', 'can authenticate as', 'member of', 'contains') | make-graph SourceNodeId --> TargetNodeId with (ExposureGraphNodes | project NodeId, NodeName, NodeLabel, NodeProperties) on NodeId // Look for existing paths between source nodes and target nodes with less than predefined number of hops | graph-match (s)-[e*1..6]->(t) where (s.NodeLabel in ('microsoft.sql/servers', 'microsoft.sql/managedinstances') and isnotnull(s.NodeProperties.rawData.allowsBasicAuth) and t.NodeLabel == 'microsoft.keyvault/vaults') project SourceName = s.NodeName , SourceType = s.NodeLabel , SourceId = s.NodeId , SourceExposedToInternet = s.NodeProperties.rawData.exposedToInternet.type , TargetName = t.NodeName , TargetType = t.NodeLabel , TargetId = t.NodeId , TargetcontainsSensitiveData = t.NodeProperties.rawData.containsSensitiveData.type , edgeIds = e.EdgeId , edgeLabels = e.EdgeLabel | extend pathLength = array_length(edgeIds) + 1

Output:

We can wrap up this logic in a generic function XGraph_PathExploration that allows to find and explore all relevant paths between source and target nodes, filtered by relevant types and properties.

This is done by changing the following required parameters in array format: sourceTypes, sourceProperties, targetTypes, targetProperties.

The following parameters have default values and are optional: maxPathLength controls the maximum length of found paths (default value 6) and resultCountLimit controls that maximum number of output (default value 50000).

The function XGraph_PathExploration goes over edges defined in non-exposed edgeTypes parameter (which you can also change) and creates paths between relevant endpoint, from single hops up to length defined by maxPathLength parameter.

After creating the paths, the function exposes the endpoints and their properties, shows the full paths in FullPath field and adds the path length metric.

let XGraph_PathExploration = (sourceTypes:dynamic, sourceProperties:dynamic , targetTypes:dynamic, targetProperties:dynamic , maxPathLength:long = 6, resultCountLimit:long = 10000) { let edgeTypes = pack_array('has permissions to', 'contains', 'can authenticate as', 'can authenticate to', 'can remote interactive logon to' , 'can interactive logon to', 'can logon over the network to', 'contains', 'has role on', 'member of'); let sourceNodePropertiesFormatted = strcat('(', strcat_array(sourceProperties, '|'), ')'); let targetNodePropertiesFormatted = strcat('(', strcat_array(targetProperties, '|'), ')'); let nodes = ( ExposureGraphNodes | project NodeId, NodeName, NodeLabel , SourcePropertiesExtracted = iff(sourceProperties != "[\"\"]", extract_all(sourceNodePropertiesFormatted, tostring(NodeProperties)), pack_array('')) , TargetPropertiesExtracted = iff(targetProperties != "[\"\"]", extract_all(targetNodePropertiesFormatted, tostring(NodeProperties)), pack_array('')) , criticalityLevel = toint(NodeProperties.rawData.criticalityLevel.criticalityLevel) | mv-apply SourcePropertiesExtracted, TargetPropertiesExtracted on ( summarize SourcePropertiesExtracted = make_set_if(SourcePropertiesExtracted, isnotempty(SourcePropertiesExtracted)) , TargetPropertiesExtracted = make_set_if(TargetPropertiesExtracted, isnotempty(TargetPropertiesExtracted)) ) | extend CountSourceProperties = coalesce(array_length(SourcePropertiesExtracted), 0) , CountTargetProperties = coalesce(array_length(TargetPropertiesExtracted), 0) | extend SourceRelevancyByLabel = iff(NodeLabel in (sourceTypes) or sourceTypes == "[\"\"]", 1, 0) , TargetRelevancyByLabel = iff(NodeLabel in (targetTypes) or targetTypes == "[\"\"]", 1, 0) , SourceRelevancyByProperties = iff(CountSourceProperties > 0 or sourceProperties == "[\"\"]", 1, 0) , TargetRelevancyByProperties = iff(CountTargetProperties > 0 or targetProperties == "[\"\"]", 1, 0) | extend SourceRelevancy = iff(SourceRelevancyByLabel == 1 and SourceRelevancyByProperties == 1, 1, 0) , TargetRelevancy = iff(TargetRelevancyByLabel == 1 and TargetRelevancyByProperties == 1, 1, 0) ); let edges = ( ExposureGraphEdges | where EdgeLabel in (edgeTypes) | project EdgeId, EdgeLabel, SourceNodeId, SourceNodeName, SourceNodeLabel, TargetNodeId, TargetNodeName, TargetNodeLabel ); let paths = ( edges // Build the graph from all the nodes and edges and enrich it with node data (properties) | make-graph SourceNodeId --> TargetNodeId with nodes on NodeId // Look for existing paths between source nodes and target nodes with up to predefined number of hops | graph-match (s)-[e*1..maxPathLength]->(t) // Filter only by paths with relevant sources and targets - filtered by node types and properties where (s.SourceRelevancy == 1 and t.TargetRelevancy == 1) project SourceName = s.NodeName , SourceType = s.NodeLabel , SourceId = s.NodeId , SourceProperties = s.SourcePropertiesExtracted , CountSourceProperties = s.CountSourceProperties , SourceRelevancy = s.SourceRelevancy , TargetName = t.NodeName , TargetType = t.NodeLabel , TargetId = t.NodeId , TargetProperties = t.TargetPropertiesExtracted , CountTargetProperties = t.CountTargetProperties , TargetRelevancy = t.TargetRelevancy , EdgeLabels = e.EdgeLabel , EdgeIds = e.EdgeId , EdgeAllTargetIds = e.TargetNodeId , EdgeAllTargetNames = e.TargetNodeId , EdgeAllTargetTypes = e.TargetNodeLabel | extend PathLength = array_length(EdgeIds) + 1 , PathId = hash_md5(strcat(SourceId, strcat(EdgeIds), TargetId)) ); let relevantPaths = ( paths | extend NodesInPath = array_concat(pack_array(SourceId), EdgeAllTargetIds), NodeLabelsInPath = array_concat(pack_array(SourceType), EdgeAllTargetTypes) | extend NodesInPathList = NodesInPath // Wrap the path into meaningful format (can be tweaked as needed) | mv-expand with_itemindex = SortIndex EdgeIds to typeof(string), EdgeLabels to typeof(string) , NodesInPath to typeof(string), NodeLabelsInPath to typeof(string) | sort by PathId, SortIndex asc | extend step = strcat( iff(isnotempty(NodesInPath), strcat('(', NodeLabelsInPath, ':', NodesInPath, ')'), '') , iff(isnotempty(SourceProperties) and NodesInPath == SourceId, SourceProperties, '') , iff(isnotempty(TargetProperties) and NodesInPath == TargetId, TargetProperties, '') , iff(isnotempty(EdgeLabels), strcat('-', EdgeLabels, '->'), '')) | summarize StepSequence = make_list(step), take_any(*) by PathId // Project relevant fields | project SourceName, SourceType, SourceId, SourceProperties, CountSourceProperties, SourceRelevancy , TargetName, TargetType, TargetId, TargetProperties, CountTargetProperties, TargetRelevancy , PathId, PathLength, Path = StepSequence | top resultCountLimit by PathLength asc ); relevantPaths };

After defining this function, we can use it by providing the lists of relevant source types, source properties, target types and target properties as well as giving other values to optional parameters. If any of the required parameters is an empty array, no filtering will be applies.

For example, we can look for all paths between different compute resources that have various vulnerabilities or are exposed to the internet, to various storage assets that are either critical or contain sensitive data:

let sourceTypesList = pack_array('microsoft.compute/virtualmachines', 'compute.instances', 'ec2.instance'); let sourcePropertiesList = pack_array('vulnerableToPrivilegeEscalation', 'vulnerableToRCE', 'hasHighSeverityVulnerabilities', 'exposedToInternet'); let targetTypesList = pack_array('microsoft.sql/servers', 's3.bucket', 'rds.db', 'storage.buckets', 'microsoft.storage/storageaccounts', 'rds.snapshot', 'microsoft.documentdb/databaseaccounts'); let targetPropertiesList = pack_array('criticalityLevel', 'containsSensitiveData'); XGraph_PathExploration(sourceTypes=sourceTypesList, sourceProperties=sourcePropertiesList , targetTypes=targetTypesList, targetProperties=targetPropertiesList)

Output:

Note that the FullPath field contains the full description of the path, with node and edge types and properties, for example:

(microsoft.compute/virtualmachines:ffcbc)[exposedToInternet]-can authenticate as->(managedidentity:23e7)-has role on->(microsoft.sql/servers:3d5c)[criticalityLevel]

This shows how the endpoints are connected, and can be used to find the proper disruption method (e.g., removing Managed Identity connecting exposed VM and critical SQL server).

Alternatively, we can look for all assets that allow public access or exposed to Internet (without specifying source type) to all keyvaults (without specifying target properties):

let sourceTypesList = pack_array(''); let sourcePropertiesList = pack_array('allowsPublicAccess', 'exposedToInternet'); let targetTypesList = pack_array('microsoft.keyvault/vaults'); let targetPropertiesList = pack_array(''); XGraph_PathExploration(sourceTypes=sourceTypesList, sourceProperties=sourcePropertiesList , targetTypes=targetTypesList, targetProperties=targetPropertiesList)

You can also wrap the XGraph_PathExploration function in a specific function with predefined parameters and use it directly for commonly used scenarios. For example, the first scenario in this section can be covered by the following function:

let XGraph_VulnerableOrExposedVMsToCriticalOrSensitiveStorage = () { let sourceTypesList = pack_array('microsoft.compute/virtualmachines', 'compute.instances', 'ec2.instance'); let sourcePropertiesList = pack_array('vulnerableToPrivilegeEscalation', 'vulnerableToRCE', 'hasHighSeverityVulnerabilities', 'exposedToInternet'); let targetTypesList = pack_array('microsoft.sql/servers', 's3.bucket', 'rds.db', 'storage.buckets', 'microsoft.storage/storageaccounts', 'rds.snapshot', 'microsoft.documentdb/databaseaccounts'); let targetPropertiesList = pack_array('criticalityLevel', 'containsSensitiveData'); XGraph_PathExploration(sourceTypes=sourceTypesList, sourceProperties=sourcePropertiesList , targetTypes=targetTypesList, targetProperties=targetPropertiesList) };

Usage -

vulnerableOrExposedVMsToCriticalOrSensitiveStorage()

Mastering Security Posture with Microsoft’s Advanced Exposure Management Tables

In this post, we delve into the core components of Microsoft Security Exposure Management - the tables ExposureGraphNodes and ExposureGraphEdges and the graph toolset for exploring them. We explain the schemas and illustrate how these tables improve the investigation of security posture by several real-world scenarios. We also present several generic queries that can be adapted to your usage by specifying the parameters.

This is more than just an introduction; it’s an invitation to master the fundamental elements of these tables. We hope this will be the first step in your ‘thinking in graphs’ transformation in the security domain.

If you are having trouble accessing Advanced Hunting, please start with this guide.

Note: For full Security Exposure Management access, user roles need access to all Defender for Endpoint device groups. Users who have access restricted to specific device groups can access the Security Exposure Management attack surface map and advanced hunting schemas (ExposureGraphNodes and ExposureGraphEdges) for the device groups to which they have access.

We hope you will start exploring your Security Exposure Management Graph and integrating it into your security practice. Stay tuned for more content, as in our upcoming posts will delve even deeper, uncovering more fascinating insights and applications.