FortiGate HA on Oracle Cloud — SD-WAN Hub for 46 Branch Offices (Case Study)

A global real estate conglomerate with 46 branch offices across India needed to consolidate its entire network under a unified security and SD-WAN architecture. The client's existing infrastructure was a patchwork of legacy MPLS links, standalone firewalls at each site, and an on-premises data centre that was running out of capacity. Oracle Cloud Infrastructure was the natural destination — the organisation already ran Oracle Autonomous Transaction Database for its core ERP and property management applications. Ogma designed and deployed the complete stack: dual FortiGate HA pairs on OCI as the SD-WAN hub, spoke VCNs for production and non-production workloads, ADVPN overlay connecting all 46 branches, a disaster recovery region, and centralized management through FortiManager and FortiAnalyzer.

This case study walks through the architecture, design decisions, deployment timeline, and lessons learned.

The Challenge

The client faced a set of problems that are common across large Indian enterprises with distributed operations:

• 46 offices with disparate security postures — each branch had its own firewall, its own ruleset, and its own local admin making changes. There was no standardised policy across the organisation
• Legacy MPLS with no application awareness — all traffic, whether business-critical Oracle APEX sessions or casual web browsing, traversed the same MPLS circuits with identical priority. No application-aware routing, no SLA monitoring, no automatic failover
• On-premises data centre at capacity — the corporate headquarters data centre had exhausted its power and rack space. Expanding on-prem would require a 12–18 month build-out with significant capital expenditure
• No centralized visibility — the security team could not see traffic patterns, threat events, or policy violations across all 46 sites from a single console. Incident investigation meant logging into individual firewalls at each location
• DR was an afterthought — disaster recovery existed as a documented plan but had never been tested. There was no automated failover, no replicated infrastructure, and no defined RPO/RTO targets
• Remote access on legacy VPN — field staff and work-from-home employees used a traditional SSL VPN that granted full Layer 3 network access once authenticated. No device posture checks, no per-application access control, no continuous verification

Architecture Design — Dual-Zone Hub on OCI

The core of the architecture is a hub VCN (Virtual Cloud Network) in the primary OCI region. This hub hosts two independent FortiGate VM pairs in active-passive high availability, each serving a distinct security function.

External FortiGate HA Pair

The external pair sits between the OCI Internet Gateway and the internal network. It handles all north-south traffic — internet-bound traffic from spoke VCNs, inbound connections from branch SD-WAN tunnels, and VPN termination for remote access users. This pair has interfaces in the untrust subnet (internet-facing) and routes inspected traffic to the trust subnet. Every IPsec tunnel from the 46 branch FortiGates terminates on this external pair.

Internal FortiGate HA Pair

The internal pair handles all east-west traffic — communication between spoke VCNs (production to non-production), spoke-to-on-premises traffic, and inter-application flows within the OCI environment. This pair connects to the trust subnet and peers with spoke VCNs via OCI Local Peering Gateways (LPGs). All traffic between spokes must traverse this internal pair for full NGFW inspection.

Why Two Pairs Instead of One

A single FortiGate pair could technically handle both north-south and east-west traffic. The decision to deploy two independent pairs was driven by three factors:

1. Separation of duties — the external pair can be sized and licensed for internet edge workloads (high IPsec throughput, SSL inspection, IPS), while the internal pair is optimised for east-west segmentation (application control, microsegmentation policies)
2. Independent scaling — if branch count grows from 46 to 100+, the external pair can be upgraded to a larger VM instance class without touching the internal pair. Conversely, adding more spoke VCNs increases east-west load without affecting the SD-WAN hub capacity
3. Blast radius containment — a misconfiguration or failure on the external pair does not disrupt east-west traffic between production workloads. Each pair fails independently within its own HA cluster

OCI Network Components

The hub VCN uses standard OCI networking constructs:

• VCN (Virtual Cloud Network) — the overarching network boundary for the hub
• Subnets — untrust (internet-facing, external FG pair), trust (internal network, internal FG pair), heartbeat (HA synchronization, dedicated per pair), and management (FortiGate admin access)
• Internet Gateway — attached to the untrust subnet for internet ingress/egress
• Route tables — custom route tables directing spoke-bound traffic through the internal FortiGate pair and internet-bound traffic through the external pair
• Network Security Groups (NSGs) — OCI-native stateful security rules as a secondary enforcement layer alongside FortiGate policies

Spoke VCN Design

The client's OCI workloads are separated into two spoke VCNs, each connected to the hub via OCI Local Peering Gateways (LPGs).

Production Spoke VCN

Hosts the production instances of Oracle APEX (the client's property management application) and FDMI (financial data management). Application servers sit in private subnets with no direct internet access. All ingress traffic arrives through the hub's external FortiGate pair and is inspected by OCI WAF before reaching the application tier. The database layer uses Oracle Autonomous Transaction Database with automatic scaling and patching. OCI Application Web Dispatcher (AWD) handles load balancing across application instances.

Non-Production Spoke VCN

Mirrors the production architecture for development and testing workloads. Separate APEX and FDMI instances run against their own Autonomous Transaction Database. The non-production VCN has its own LPG to the hub, ensuring dev/test traffic is segmented from production at the network level. Traffic between production and non-production must traverse the internal FortiGate pair, where strict policies prevent non-production workloads from accessing production databases or services.

LPG Peering Architecture

Each spoke VCN peers with the hub VCN through a dedicated LPG. Route tables in the spoke VCNs set the default gateway to the hub's internal FortiGate pair via the LPG. This means all traffic leaving a spoke — whether destined for another spoke, the internet, a branch office, or the on-premises data centre — is inspected by the internal FortiGate. There is no direct spoke-to-spoke communication that bypasses the firewall.

High Availability Design

FortiGate HA on OCI uses FGCP (FortiGate Clustering Protocol) in active-passive mode. The design accounts for OCI-specific constraints that differ from on-premises HA deployments.

Unicast Heartbeat

OCI does not support multicast traffic within VCNs. FortiGate HA heartbeat communication, which uses multicast by default, must be configured for unicast mode. A dedicated heartbeat subnet is provisioned within the hub VCN, with a point-to-point link between the primary and secondary FortiGate VMs. The heartbeat carries configuration synchronization, session table replication, and health monitoring.

VNIC Failover

When the active FortiGate fails, the passive unit must assume the active unit's IP addresses to maintain traffic flow. On OCI, this is achieved through secondary VNIC migration. The FortiGate HA configuration includes OCI SDN connector integration that uses the OCI API to reassign secondary private IPs and update route table next-hop targets from the failed active to the newly promoted standby. This API-driven failover adds a small overhead compared to gratuitous ARP on bare metal, but the total failover time was validated at under 10 seconds during pre-production testing.

Independent HA Clusters

Both the external and internal FortiGate pairs operate as independent HA clusters. If the active external FortiGate fails, it fails over to its passive partner. The internal pair continues operating without interruption, and vice versa. This ensures that a single hardware or software failure does not impact both north-south and east-west traffic simultaneously. Each pair has its own heartbeat subnet, its own OCI SDN connector credentials, and its own failover logic.

SD-WAN Overlay — 46 Branches to Cloud Hub

With the OCI hub infrastructure in place, the next layer is the SD-WAN overlay connecting all 46 branch offices to the cloud.

IPsec Tunnels to Hub

Each branch FortiGate establishes IPsec tunnels to the hub's external FortiGate HA pair in the primary OCI region. Branches with dual WAN links (the majority of the 46 sites) build two tunnels — one per WAN link — giving each branch two independent paths to the cloud hub. SD-WAN performance SLA monitors both tunnels continuously for latency, jitter, and packet loss, steering application traffic to the best-performing path in real time.

ADVPN for Dynamic Spoke-to-Spoke

ADVPN (Auto Discovery VPN) eliminates the need for a full mesh of branch-to-branch tunnels. In the default hub-and-spoke topology, all inter-branch traffic passes through the OCI hub. When two branches need direct communication — for example, a regional office transferring large files to a nearby satellite office — ADVPN dynamically negotiates a direct spoke-to-spoke IPsec tunnel. The tunnel is established on demand, maintained as long as traffic flows, and torn down when idle. This keeps the hub from becoming a bottleneck for lateral branch traffic while maintaining the simplicity of a hub-and-spoke configuration.

BGP over IPsec

Route exchange between branches and the hub uses BGP (Border Gateway Protocol) running over the IPsec overlay tunnels. Each branch advertises its local subnets to the hub, and the hub advertises OCI spoke VCN subnets and on-premises routes back to the branches. BGP provides automatic convergence when links fail or new routes appear, and BGP communities are used to tag routes by region and branch type for granular policy control.

Application-Aware Steering

The SD-WAN configuration uses FortiOS performance SLA rules to steer traffic based on application identity:

• Business-critical applications (Oracle APEX, FDMI, ERP) — steered over the primary WAN link with strict latency and packet-loss SLAs. Failover to secondary link if primary degrades below threshold
• Voice and video — steered based on jitter and MOS (Mean Opinion Score) thresholds, with priority queuing enabled
• General internet traffic — steered over the lowest-cost link that meets a basic quality bar, enabling direct internet breakout at the branch with full NGFW inspection

FortiManager Templates at Scale

Managing 46 branch FortiGates individually is not sustainable. FortiManager templates are the backbone of this deployment:

• One SD-WAN template — defines overlay tunnels, performance SLA monitors, and steering rules. Applied to all 46 branches with per-device variables (WAN interface names, public IPs) handled by FortiManager's metadata system
• One security policy set — unified firewall policy covering application control, IPS, web filtering, and SSL inspection. Pushed to all branches from a single policy package
• Zero-touch provisioning (ZTP) — new branch FortiGates are shipped, plugged in by local staff, and self-configure by contacting FortiManager via FortiCloud. During the rollout, all 46 branches were provisioned in a single week using this method

Branch FortiGate models were selected based on office size. Small offices with under 20 users received desktop-form-factor units. Mid-size offices with 20–100 users received higher-throughput desktop models. Regional hubs with 100–200+ users received rack-mount units with SP5 or NP7 ASICs for line-rate NGFW inspection.

Security Stack

The security architecture layers OCI-native services with FortiGate capabilities to create defence in depth across the entire environment.

OCI-Native Security Services

• OCI Vault — centralized certificate and secret management for the FortiGate VMs, application servers, and database connections. No credentials stored in configuration files
• OCI Key Management — encryption key lifecycle management for data at rest across spoke VCNs. Customer-managed keys rotate automatically on a defined schedule
• OCI WAF — deployed in front of the production spoke VCN's web applications. Handles OWASP top-10 protection, bot management, and rate limiting before traffic reaches the FortiGate inspection layer
• OCI Logging — VCN flow logs, audit logs, and OCI service events are collected and forwarded to FortiAnalyzer for correlation with FortiGate security logs. This provides a unified audit trail across cloud-native and firewall-generated events

FortiGate Security Policies

All traffic traversing the FortiGate pairs — north-south and east-west — is subject to full NGFW inspection:

• IPS (Intrusion Prevention System) — signature and anomaly-based detection on all spoke traffic
• Application Control — granular application identification and enforcement. Oracle APEX and FDMI traffic is identified by application signatures, not just port numbers
• Web Filtering — URL and content filtering on all internet-bound traffic from spokes and branches
• SSL Inspection — deep packet inspection on encrypted traffic flowing between spokes, with certificate exemptions for banking and healthcare applications as required by compliance
• Active Directory Integration — FortiGate LDAP integration with the client's on-premises AD enables identity-based policies. Firewall rules reference AD security groups rather than IP addresses, so policies follow users regardless of which branch or VPN they connect from

FortiAnalyzer for Centralized Logging

FortiAnalyzer aggregates logs from all FortiGate instances — the four OCI hub VMs and all 46 branch appliances. The security team uses a single dashboard for threat detection, traffic analytics, SD-WAN health monitoring, and compliance reporting. OCI Logging data is ingested via syslog forwarding, giving the SOC team a correlated view of cloud infrastructure events alongside firewall security events.

Disaster Recovery

The DR architecture mirrors the primary region's hub design in a geographically separate OCI region.

DR Region Infrastructure

• Hub VCN — identical untrust and trust subnets with a FortiGate HA pair pre-configured and on standby
• Spoke VCN — DR application instances and Autonomous Transaction Database with Oracle Data Guard replication from the primary region
• FortiGate configuration — synced from FortiManager, ensuring the DR FortiGates have identical policies, SD-WAN templates, and security profiles as the primary

Failover Mechanism

If the primary OCI region becomes unavailable, the SD-WAN branches fail over to the DR hub automatically. This works through BGP route preference — branches maintain tunnels to both the primary and DR hubs simultaneously. The primary hub advertises routes with a higher BGP local preference. When primary routes are withdrawn (due to hub failure), branches automatically steer traffic to the DR hub within the BGP convergence interval.

RPO and RTO Targets

• RPO (Recovery Point Objective) — less than 1 hour for database workloads via Oracle Data Guard asynchronous replication
• RTO (Recovery Time Objective) — less than 4 hours for full DR activation, including database switchover, application startup, and DNS updates
• Testing cadence — DR failover is tested quarterly with a documented runbook. Each test validates database switchover, FortiGate SD-WAN reconvergence, application availability, and client access from branch offices

Remote Access

The client's remote workforce connects to the OCI-hosted applications through two methods, depending on the security posture requirements.

Traditional VPN

For users who need broad network access (IT administrators, infrastructure engineers), SSL VPN terminates on the hub's external FortiGate pair. Authentication is performed against Active Directory with two-factor verification. VPN users are placed into specific VPN-only zones with firewall policies that restrict lateral movement to only the resources required for their role.

Zero-Trust Remote Access (Recommended)

For the majority of remote users, the recommended approach is Dast Connect for zero-trust remote access. Instead of granting full network access via VPN, Dast Connect provides per-application tunnels with continuous device posture checks:

• Per-application access — users are granted access to specific applications (Oracle APEX, FDMI, internal portals) rather than network segments. No lateral movement is possible because users never have IP-level connectivity to the underlying network
• Device posture verification — endpoint health is verified before and during every session. OS patch level, disk encryption status, antivirus state, and domain membership are checked continuously
• Active Directory authentication — all remote sessions authenticate against the client's AD infrastructure. Conditional access policies enforce additional checks based on user group, device type, and connection location

Deployment Timeline and Results

The deployment was executed in three intensive weeks with parallel workstreams.

Week 1: OCI Infrastructure + Spoke VCNs

OCI compartment and VCN provisioning, subnet design, route table configuration, NSG rules. FortiGate VM deployment and HA configuration for both external and internal pairs. OCI SDN connector integration and failover validation. FortiManager and FortiAnalyzer deployment. In parallel, production and non-production spoke VCNs were deployed with LPG peering to the hub. Oracle Autonomous Transaction Database provisioning and OCI WAF deployment began simultaneously. By end of Week 1, the entire OCI hub-spoke infrastructure was operational and validated.

Week 2: Branch SD-WAN Rollout (All 46 Sites)

This was the most operationally intensive phase. FortiManager SD-WAN and security templates were finalised and validated against a pilot group of 4 branches on Day 1. Once validated, the remaining 42 branches were onboarded aggressively using FortiManager zero-touch provisioning — FortiGate appliances had been pre-shipped to all sites the previous week. Local staff plugged in the hardware, and each unit self-configured against FortiManager. The Ogma deployment team ran parallel validation across multiple sites simultaneously — tunnel status, SD-WAN SLA health, security policy enforcement, and application reachability — clearing 15–20 branches per day. All 46 branches were production-ready by end of Week 2.

Week 3: DR Region + Application Migration + Go-Live

DR region hub VCN and FortiGate HA pair deployment. Spoke VCN with Oracle Data Guard replication from primary. Branch SD-WAN configuration updated to include DR hub tunnels with lower BGP preference. Full DR failover test executed and documented. In parallel, Oracle APEX and FDMI workloads were migrated from on-premises to OCI spoke VCNs. Final security policy hardening, IPS signature updates, and application performance baseline completed. Production go-live at end of Week 3.

Results

• Single-pane visibility — FortiAnalyzer provides a unified dashboard across all 46 branch FortiGates and 4 OCI hub VMs. The security team can investigate any event, at any site, from one console
• WAN cost reduction — replacing legacy MPLS circuits with dual internet links at each branch significantly reduced monthly WAN spend. SD-WAN application steering ensures business-critical traffic still receives priority treatment
• Sub-second application response — Oracle APEX and FDMI users at branch offices experience sub-second response times to the OCI-hosted applications, compared to multi-second response times over the previous MPLS-to-on-premises path
• Zero security incidents — in the six months following deployment, the organisation recorded zero security incidents attributable to the new infrastructure. The centralised NGFW policy enforcement and SD-WAN segmentation eliminated the inconsistent security posture that had previously existed across sites
• Tested DR — DR failover has been tested quarterly since deployment, with full branch reconvergence to the DR hub validated each time

Key Lessons Learned

1. OCI's flat networking model simplifies hub-spoke — VCN + LPG peering is conceptually simpler than AWS Transit Gateway or Azure Virtual WAN. Route tables are straightforward, and LPG bandwidth is not a bottleneck. For organisations already running Oracle database workloads, OCI is the natural cloud platform and the networking constructs are well-suited to FortiGate hub-spoke architectures
2. Unicast heartbeat is mandatory — FortiGate HA on OCI requires unicast heartbeat configuration since OCI VCNs do not support multicast. This is a known requirement documented by both Fortinet and Oracle, but it is easy to overlook if your team is accustomed to on-premises HA deployments where multicast works out of the box
3. ADVPN dramatically reduces tunnel count — with 46 branches, a full mesh would require over 1,000 static tunnels. ADVPN reduces this to 46 hub-and-spoke tunnels (92 with dual WAN), with spoke-to-spoke tunnels created dynamically on demand. The operational simplification is significant
4. FortiManager templates are essential at scale — managing 46+ FortiGates without centralized templates is not sustainable. Every branch configuration change, security policy update, and SD-WAN rule modification is made once in FortiManager and pushed to all devices. Manual, per-device configuration was eliminated entirely
5. Build DR on day one — the client's previous infrastructure had no tested DR. Building the DR region in Week 3 alongside application migration meant it was ready at go-live, not retrofitted months later. This added no extra time to the project because it ran in parallel with workload migration
6. OCI SDN connector failover adds latency — FortiGate HA failover on OCI relies on API calls to reassign VNICs and update route tables. This is slower than on-premises gratuitous ARP failover. Sub-10-second failover is achievable with proper tuning, but the team should account for this during HA testing and set realistic expectations with stakeholders

Why Ogma

Ogma Consulting designed, deployed, and manages this entire infrastructure. Our team brings:

• NSE7-certified FortiGate engineers — deep expertise in FortiGate HA, SD-WAN, ADVPN, and FortiManager at scale
• OCI deployment experience — not just AWS and Azure. We understand OCI's networking model, SDN connector integration, and the specific requirements for running FortiGate VMs in OCI
• End-to-end delivery — from architecture design through deployment to ongoing managed services. The client did not need to engage separate vendors for cloud, security, and SD-WAN
• India-based team with pan-India reach — the 46-branch rollout required coordinating with local ISPs, shipping hardware, and validating connectivity across multiple cities and tiers. Our India-based operations team managed this end to end

Planning a similar FortiGate SD-WAN deployment on OCI or another cloud platform? Contact Ogma for a technical consultation.

Frequently Asked Questions