Tools:

Adversary Emulation Platform:

Reporting Platform:

Philosophy

Full-stack vs Assumed Breach.

I decided that we will adopt an assumed-breach scenario in the earlier maturity phases of the BAS program. This means that we will setup the environment to mimic a compromised environment where a threat actor had already established an initial foothold and deployed a C2 beacon (Caldera agent).

As the BAS program matures, we can work with full-stack scenarios where the BAS exercises will comprise of entire steps of the MITRE ATT&CK Framework. This means that the exercises will include things like external recon, phishing campaigns, and other tactics carried by threat actors before they get an initial foothold in the environment.

Methodology:

Approach #1:

SRA or Security Risk Advisors, the people behind Vectr, offer Industry Threat Indexes and Benchmarks and one of these indexes is the Financial Threat Index which comprises of 50 Procedures (Attacks) that map to the MITRE ATT&CK framework.

The idea is that my company can use this FS Index in Vectr, craft attack simulations in MITRE’s Caldera that map directly to it and conduct the tests.

SRA’s FS Threat Index is used by many members of the Financial Services Information Sharing and Analysis Center.

Approach #2:

The second approach is more CTI driven and depends on the Security staff digesting multiple Cyber Threat Intel streams and crafting focused BAS scenarios based on that.

Some of the CTI sources would be BlueVoyant, Microsoft’s threat analytics, Palo’s Unit 42, CISA, MITRE, Red Canary and other free or paid CTI streams.

An example would be: Us learning about a similar financial institution being the victim of a Ransomware attack carried out by a specific APT using one or more of these CTI streams. The next would be to research the corresponding TTP of the specific APT and craft a targeted BAS scenario against a subset of our organization’s devices.

Due to staff limitations and the security team members being engaged in different security projects, the preferred approach would be the 1^st one that leverages the SRA’s FS Threat Index because that minimizes the amount of leg work required before conducting BAS tests.

Demo/POC:

Created a new Environment in Vectr and attached to it the Financial Services SRA template.

Clicked on the newly created Assessment to see the list of Campaigns. (Each campaign maps to a Tactic in the MITRE ATT&CK framework.

For the sake of keeping the POC’s scope small and manageable, we are going to limit the exercise to 5 Campaigns.

Now we are going to switch gears to Caldera and create a new Adversary with the same name as our Vectr Assessment, “2024 Q3: Financial Services Threat Index”.

Currently our new Adversary Profile does not have any Abilities. Abilities can be directly translated into MITRE ATT&CK TTPs.
In Vectr, open the Discovery Campaign and observe the different Techniques and Test Cases.

Clicked on the first Test Case in the table and reviewed the Red Team details.

We see that the Test Case “Domain Controller discovery via nltest” directly maps to ATT&CK Technique T1018 – Remote System Discovery.
Next, we are going back to Caldera, and under our “2024 Q3: Financial Services Threat Index” Adversary Profile, we are going to add a new ability, and we are going to search for “Remote System Discovery – nltest” and select it from the list of over 1600 abilities that Caldera already comes with.

Now we are going to do the same for all the Test Cases within our 5 Vectr Campaigns. You should end up with something like this: (Remember to SAVE)

Caldera Agents:

The Caldera platform is agent-based, meaning that the target machines require an agent on them that is going to facilitate running the different attacks/abilities we added to our Adversary Profile.

Most likely, Microsoft Defender for Endpoint is going to quarantine the .exe agent when you copy it to the target machine. We need to make sure we whitelist the agents based on hash and file exclusions.

In Caldera, navigate to the Agents section and click Deploy an agent.
Select the Sandcat default agent and pick Windows. Then enter the IP address of the Caldera servers and you will be presented with a PowerShell command that you need to run on the target machine to install the agent.

Copy the PS command and run it in an elevated PowerShell session. In this POC, we are going to install the agent on 1 Domain Controller, and 2 Windows 10 machines.
Under Agents, you should now see the different machines you installed the agents on.

Operation (Running the actual BAS exercise)

In this next step, we are going to tie everything together using an Operation, the Adversary profile (with its abilities), and the target machines where we installed our agents.

Click Operations and select New operation.

We are going to call this new Operation “2024 Q3: Financial Services Threat Index Operation”.
Fill out the rest of the settings like the screenshot below:

Click Start to begin the BAS. You should see the steps being performed one at a time. And you also could drop in manually and start a manual command.

While the tests are running, switch back to Vectr and update the campaigns to In Progress.

Ideally, we only want to run these tests Autonomously if we tweaked everything and we are sure we are not going to run into issues. A more effective way, but time consuming, is to set Caldera to run the test manually, where the operator will manually trigger the next attack. This will give him the chance to update Vectr accordingly as the test is running and get accurate info as much as possible.
Next, “Blue Team” to review the abilities/techniques one at a time and update the Test Cases in Vectr.
Once you are done, you should end up with something like this:

Reports

Now that our BAS exercise is completed by both the red and blue teams, we can start generating reports and have a better understanding of our defense coverage and areas where we should improve. For this, Vectr’s powerful reporting capabilities are going to come in handy.

We have different report types under the Reporting tab.

Below are a few examples:

Threat Index Scoring

MITRE ATT&CK Alignment

Test Case Drilldown

Enhancing Logging, Detections and Preventions

The BAS reports should help the org assess which MITRE ATT&CK Tactics are lacking coverage and protection.

Findings should be leveraged to:

Ingest missing logs into the SIEM solution.
Create/Enable Detection rules, per MITRE recommendations, to increase coverage and harden defenses.

Continuous Testing and Progress Tracking

The next step in the BAS maturity program would be to continuously run the BAS exercises on a set schedule and compare results with previous exercises.

For the same of the Demo/POC, we are going to Clone the Assessment we have gone through and rename it to “2024 Q4: Financial Services Threat Index”, and then we are going to populate it with different results/outcomes so we can do a comparison in the Reports.