Introduction
AC3 are a secure managed services provider focused on building, running and security mission critical environments.
We are also longstanding Palo Alto Network partner, using Palo Alto PanOS and GlobalProtect for our customer environments and for managing access for select technical environments.
With that comes extensive knowledge of the security platform, along with deep industry connections and technological expertise.
Recently, AC3 engineers leveraged this knowledge to assist Palo Alto in identifying and resolving some vulnerabilities in the Palo Alto product suite.
Problem Identification
Following routine firewall and VPN gateway patching, some AC3 technical staff reported being unable to connect to the Palo Alto GlobalProtect VPN. This was immediately investigated by engineering teams.
When attempting to authenticate, the process appeared to proceed normally up to the point where the SAML response was sent back to the VPN. At this point, users encountered a “Temporary Unavailable” page, signalling a deeper issue.
Initially, the team suspected a regression or incompatibility with some VPN clients. To eliminate this possibility, AC3 engineers replicated the issue while monitoring logs on the firewall.
Diving into the Logs
By observing the logs on the firewall (sslvpn-access.log), the following error was captured during an authentication attempt:
runtime error: invalid memory address or nil pointer dereference, stack: goroutine 1678733 [running]:
main.(*GpTask).RunHttp.func2.1(0xc007d68a00, 0xc00c03a660)
/opt/build/bamboo-agent-home-4/xml-data/build-dir/LA-GPSVC71-JOB1/build/src/apps/pan_gpsvc_task.go:746 +0x263
panic(0xbe20a0, 0x114dd70)
/usr/local/go/src/runtime/panic.go:679 +0x1e0
main.gpGenerateSessionCookie(0xc007d68a00, 0xc00615d160, 0xe, 0x0, 0x0, 0x0, 0x0)
/opt/build/bamboo-agent-home-4/xml-data/build-dir/LA-GPSVC71-JOB1/build/src/apps/pan_gpsvc_handler.go:1186 +0x100
main.samlAcs(0xc007d68a00, 0x0, 0x0)
/opt/build/bamboo-agent-home-4/xml-data/build-dir/LA-GPSVC71-JOB1/build/src/apps/pan_saml_acs.go:220 +0x1dc0
main.(*GpTask).RunHttp.func2(0xc007d68a00, 0xc00c03a660)
/opt/build/bamboo-agent-home-4/xml-data/build-dir/LA-GPSVC71-JOB1/build/src/apps/pan_gpsvc_task.go:762 +0x2cb
created by main.(*GpTask).RunHttp
/opt/build/bamboo-agent-home-4/xml-data/build-dir/LA-GPSVC71-JOB1/build/src/apps/pan_gpsvc_task.go:740 +0x315
0, taskid 194993, user xxxx
The error revealed a “nil pointer dereference” in the firewall’s SSL VPN service, suggesting a backend processing issue rather than a problem with the client frontend. This finding was critical as it shifted focus from debugging our custom code to investigating potential issues on the firewall itself.
Reproducing the Issue
To better understand the problem, we attempted to reproduce it using direct API calls. First, we extracted the SAML request from the pre-login URL using curl:
curl -s "https://xxxx/ssl-vpn/prelogin.esp?tmp=tmp&clientVersion=5800&clientos=macos" | grep -oP '(?<=).*?(?= )' | base64 -d
Decoding the Base64-encoded SAML request revealed an Azure/Office365 URL, which was embedded in the XML response.
When this URL was accessed in a browser, authentication prompts were completed successfully, and completion of the authentication the IDP then posted a SAML response back to the VPN Gateway.
However, this triggered the same error when returning a response (where we expected a pre-login cookie) and every subsequent request reproduced the same “nil pointer dereference” issue, confirming that the bug was consistently repeatable.
Analysis and Escalation
The repeated occurrence of the error raised questions about whether this was a bug introduced during the firewall patch or an underlying security vulnerability. Given the potential implications, the issue was escalated to Palo Alto Networks.
Upon reviewing the logs, providing steps to reproduce, and discussing the findings, the Palo Alto security team confirmed it was a software flaw. Palo Alto advised that this issue would be addressed via a published CVE, identified as CVE-2024-2550.
This CVE highlights a critical vulnerability related to how SAML authentication responses are processed by their GlobalProtect VPN.
Lessons Learned and Next Steps
The incident was a good reminder of the complexities inherent in managing custom integrations with enterprise security products. Through thorough investigation and collaboration, we were able to identify the root cause and ensure its resolution in the broader security ecosystem.
