Menu
Encrypting web traffic has long been commonplace to ensure confidentiality by preventing eavesdropping during transmission. Likewise, requiring users to authenticate before they are permitted to access web resources – and to require users to prove that, further, they are explicitly authorized to access a resource – are widely recognised security techniques. However, requiring users to prove that the message received by the server has not been tampered with – that is, guaranteeing its integrity – is a consideration that has only recently started to gain widespread traction.
In this blog post, we delve into the complexities surrounding one such data integrity technique: HTTP message signing. We explore the different specifications currently in use, dissecting their mechanisms and operations. Additionally, we examine the challenges the techniques may present when scanning web applications and discuss how AppCheck has solved these problems to integrate message signing support into our vulnerability scanner.
Message signing provides a robust solution for guaranteeing the integrity and authenticity of messages over a communication protocol, regardless of the number of intermediaries or the complexity of the network architecture. It ensures that even if messages traverse multiple network hops or pass through external inspection points, their integrity remains intact. This approach offers a higher level of security assurance compared to traditional transport layer-based mechanisms, making it particularly valuable in environments where data integrity and trustworthiness are paramount.
At its core, message signing involves digitally signing each message before transmission using cryptographic techniques. This signature, attached to the message, serves as a unique identifier, and can be validated to confirm that the message has not been altered during transit. Upon receipt, the recipient can verify the signature, using an agreed method, confirming the message’s integrity and authenticity. This fundamental principle of message level signing provides a structure for safeguarding messages transmitted across untrusted networks such as the internet and protecting against potential message tampering by malicious actors.
The burden of ensuring message integrity has long been delegated to the transport layer over which the application is operating over, i.e. Transport Layer Security (TLS). Initially this was acceptable because traditionally the TLS connection terminated at the application or at least within the boundary of the organisation responsible for the application. However, TLS only protects the message through a single connection, not across multiple hops between the client and the application. With the increased prevalence of TLS-terminating gateways and proxies, TLS inspection appliances or services outside of the application organisation’s control (such as Intrusion Detection System (IDS) / Intrusion Prevention System (IPS)), and the adoption of cloud “serverless” technologies; the risk of a message being modified by malicious third parties with access to the network traffic has increased in recent years. As such there is a pressing need for a solution that ensures end-to-end message integrity and authenticity. Therefore, the introduction of HTTP message signing has become a powerful technique in ensuring end-to-end message integrity and authenticity.
Despite a draft specification that has been evolving for some time (RFC9421) a definitive interoperable standard has yet to emerge. However, request signing schemes have been implemented as part of other standards (such as X-Hub-Signature in the WebSub standard, and JSON Web Signature in RFC7515), some vendors have implemented their own techniques which are documented for client usage (for example AWS Signature Version 4), and many developers have implemented bespoke solutions which are often not publicly documented and implemented only in their own client and server-side code.
The specifics of these implementations vary considerably. One limitation in several implementations is that only the message body is signed, not any of the message properties (metadata) contained in the HTTP message headers. This means that any data requiring integrity guarantees must be inserted within the HTTP body – an approach that undermines many of the benefits and interoperability of the HTTP standard which rely on HTTP headers. A key challenge in providing a message integrity scheme for HTTP is the common and benign modification of HTTP messages as they traverse intermediary systems. For example, a common modification is for a reverse proxy to add the origin IP address in the X-Forwarded-For HTTP header, and even changing the transfer encoding is not unheard of. This means that simply creating a signature of the entire HTTP request is not feasible.
A common approach to HTTP message signing is to initially create a string representing the request (“canonical request”), including all of the elements that require integrity: this typically includes the request method, URL, parameters or body, and a subset of the HTTP headers. The canonical request string is then hashed and signed using a secret key and the resulting signature is then attached to the HTTP message in a HTTP header (typically the Authorization header) or a parameter. Of course, the application must be able to form the exact same canonical request, which means the format of the canonical request must generally be explicitly predefined; for example, considering just the HTTP headers, at least following questions must be answered:
The signature generation algorithm must also be predefined, including the hashing algorithm and type of secret key to use.
The secret key used in message signing may either be symmetric or asymmetric depending on the requirements of the system. A symmetric key is typically a shared secret between the two parties, which the client uses to sign the message and the application uses to validate the signature. This type of key does not provide any non-repudiation since either party can use it to create a signature for the message. If non-repudiation is required, then an asymmetric key must be used instead: the private key is only held by the client here and is used to sign the message, but the public key can be used by anyone to confirm that the message was signed with the associated private key. Asymmetric cryptography is much less accessible to users since it requires secure key management and exchange practices which can be a challenging to implement.
Where and how the secret key is shared is specific to the application: the key may be pre-shared from a user interface or via email; dynamically generated for temporary use and embedded in the HTML or JavaScript returned when first visiting the application or in a JSON response after authentication; the user may be responsible for uploading a public key to a trusted store; or one of any number of other methods.
Another challenge for message integrity and authenticity that message signing schemes must often overcome is the threat presented by potential message replay attacks. If an attacker is able to capture a signed message and resend it to the application to the same effect then the signature is not fully validating the authenticity of the message – this can be enough to carry out an attacker’s malicious intent, for example deleting resources or granting permissions. A common approach to prevent such replay attacks is to include a timestamp within the request which the application can validate to ensure the request has not expired – it is critical that the field containing the timestamp is included in the canonical request (and therefore the request signature) so that the timestamp integrity is itself verifiable, in order to prevent an attacker from simply modifying the timestamp.
Vulnerability scanning web applications that utilise request signing is possible, but it does require a different approach to scanning, so support for it may be missing in some legacy vulnerability scanners. In order to provide effective coverage, a vulnerability scanner must first capture a message between the client and the application, usually via crawling the application in a headless browser and capturing messages. Legacy methods of crawling an application which only extract links from the HTML are incapable of capturing the signed messages since they are created dynamically, typically with JavaScript.
It is critical that when fuzzing for vulnerabilities, scanners are able to form valid messages which are accepted by the application with the only functionally changing area being the one that is currently under test. The captured messages are typically used as a baseline; however, they must be replayed so that responses can be compared to identify “normal” differences in response duration and content which must be used to weed out false positives that would otherwise be caused by the changes. Additionally, fuzzing involves sending many payloads in the different parts of the request processed by the application and observing differences in response and behaviour.
If the vulnerability scanner does not update the signature to account for the changes made to the request, the function of the application which the fuzzing is intending to test will remain unassessed because the application will reject the request when validating the signature before attempting to proceed with further processing.
A frequent challenge for vulnerability scanners is supporting scanning applications built with the vast array of web technologies that are being continually proliferated with varying levels of adoption and standardisation. HTTP request signing is a good example of this in action, without the existence of and adherence to a defined standard:
Recognising the increasing demand for vulnerability scanning of signed HTTP requests, AppCheck are integrating support for message signing into the authentication management and request manipulation components within the web application scanning engine through simple chain-able request processing rules.
AppCheck will be initially adding predefined rulesets for AWS v4 Signing using static credentials configured by the user or temporary credentials captured during the authentication analysis. AWS v4 Signing is a scheme required by multiple AWS services including the Amazon API Gateway which provides access to API’s deployed by developers to AWS and has seen increasing adoption as organisations move to a “cloud first” strategy.
In the near future, an interface to allow users to define their own request processing rulesets will be added to the AppCheck interface; this will facilitate scanning applications which use bespoke signing schemes or off the shelf solutions which do not yet have a predefined ruleset from AppCheck.
AppCheck is a software security vendor based in the UK, offering a leading security scanning platform that automates the discovery of security flaws within organisations websites, applications, network, and cloud infrastructure. AppCheck are authorized by the Common Vulnerabilities and Exposures (CVE) Program as a CVE Numbering Authority (CNA).
As always, if you require any more information on this topic or want to see what unexpected vulnerabilities AppCheck can pick up in your website and applications then please contact us: info@localhost
No software to download or install.
Contact us or call us 0113 887 8380
AppCheck is a software security vendor based in the UK, offering a leading security scanning platform that automates the discovery of security flaws within organisations websites, applications, network and cloud infrastructure. AppCheck are authorized by te Common Vulnerabilities and Exposures (CVE) Program aas a CVE Numbering Authority (CNA)
