Fingerprint empowers developers to stop online fraud at the source

Fingerprint

fingerprint

How to Use Fingerprint Scanners

Authors:
(1) Todd K. Moon, Electrical and Computer Engineering Department, Utah State University, Logan, Utah;
(2) Jacob H. Gunther, Electrical and Computer Engineering Department, Utah State University, Logan, Utah. Table of Links Abstract and 1 Introduction and Background 2 Statistical Parsing and Extracted Features 3 Parse Tree Features 4 Classifier 5 Dimension Reduction 6 The Federalist Papers 6.1 Sanditon 7 Conclusions, Discussion, and Future Work A. A Brief Introduction to Statistical Parsing B. Dimension Reduction: Some Mathematical Details References Abstract Over the years there has been ongoing interest in detecting authorship of a text based on statistical properties of the text, such as by using occurrence rates of noncontextual words. In previous work, these techniques have been used, for example, to determine authorship of all of The Federalist Papers. Such methods may be useful in more modern times to detect fake or AI authorship. Progress in statistical natural language parsers introduces the possibility of using grammatical structure to detect authorship. In this paper we explore a new possibility for detecting authorship using grammatical structural information extracted using a statistical natural language parser. This paper provides a proof of concept, testing author classification based on grammatical structure on a set of “proof texts,” The Federalist Papers and Sanditon which have been as test cases in previous authorship detection studies. Several features extracted from the statistical natural language parser were explored: all subtrees of some depth from any level; rooted subtrees of some depth, part of speech, and part of speech by level in the parse tree. It was found to be helpful to project the features into a lower dimensional space. Statistical experiments on these documents demonstrate that information from a statistical parser can, in fact, assist in distinguishing authors. 1 Introduction and Background There has been considerable effort over the years related to using statistical methods to identify authorship of texts, based on examples from candidate authors, in what is sometimes called “stylometry” or “author identification.” Statistical analysis of documents goes back to Augustus de Morgan in 1851 [1, p. 282], [2, p. 166], who proposed that word length statistics might be used to determine the authorship of the Pauline epistles. Stylometry was employed as early as 1901 to explore the authorship of Shakespeare [3]. Since then, it has been employed in a variety of literary studies (see, e.g. [4, 5, 6]), including twelve of The Federalist Papers which were of uncertain authorship [7] — which we re-examine here — and an unfinished novel by Jane Austen —which we also re-examine here. Information theoretic techniques have also been used more recently [8]. Earlier work in stylometry has been based on “noncontextual words,” words which do not convey the primary meaning of the text, but which act in the background of the text to provide structure and flow. Noncontextual words are at least plausible, since an author may address a variety of topics, so particular distinguishing words are not necessarily revealing of authorship. In noncontextual word studies, a set of most common words noncontextual is selected [2], and documents are represented by word counts, or ratios of word counts to document length. A review of the statistical methods is in [9]. As a variation, sets of ratios of counts of noncontextual word patterns to other word patterns are also employed [10]. Statistical analysis based on author vocabulary size vs. document length — the “vocabulary richness” — has also been explored [11]. For other related work, see [12, 13, 14, 15] A more recent paper [16] considers the effectiveness of a wide variety of feature sets. Feature sets considered there include: vectors comprising frequencies of pronouns; function words (that is, articles, pronouns, particles, expletives); part of speech (POS); most common words; syntactic features (such as noun phrase, or verb phrase); or tense (e.g. use of present or past tense); voice (active of passive). In [16], feature vectors are formed from combinations of histograms, then reduced in dimensionality using a two-stage process of principle component analysis [17] followed by dimension reduction using linear discriminant analysis (LDA). In their LDA, the within-cluster scatter matrix is singular (due to the high dimension of the feature vectors relative the number of available training vectors), so their scatter matrix is regularized. To test this, the authors consider a range of regularization parameters, selecting one which gives the best performance. More recent work [18] mentions the survey in [15] in which commonly used features in the authorship field are word and character n-grams. As noted, there are risks the statistical methods might be biased by topic-related patterns. As [18] observe, “an authorship classifier (even a seemingly good one) might end up unintentionally performing topic identification if domain-dependent features are used. ... In order to avoid this, researchers might limit their scope to features that are clearly topic-agnostic, such as function words or syntactic features.” The work presented here falls in the latter category, making use of grammatical structures statistically extracted from the text. These appear to be difficult to spoof. Examination of other recent works [19, 20] indicate that there is ongoing interest in author identification methods, but none making use of the grammatical structures use here; there is a tendency to rely more on traditional n-grams. In this work the feature vectors are obtained using tree information from parse trees from a natural language parsing tool [21]. These features were not among the features considered in [16]. The grammatical structures are, it seems, more subtle than simple counts of classes of words, and hence may be less subject to spoofing or topic bias, since it seems unlikely that an author intending to imitate another would be able to coherently track complicated patterns of usage, and the features do not include any words from the documents. It is found that the tree-based features perform better than the POS features on the test data considered. The feature vectors so obtained can be of very high dimension, so dimension reduction is also performed here. However, to deal with the singularity of the within-cluster scatter matrix, a generalized SVD approach is used, which avoids the need to select a regularization parameter. This paper provides a proof-of-concept of these tree-based features to distinguish authorship by applying them to documents which have been previously examined, The Federalist Papers and Sanditon. The ability to classify by authorship is explored for several feature vectors obtained from the parsed information. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Todd K. Moon, Electrical and Computer Engineering Department, Utah State University, Logan, Utah; (2) Jacob H. Gunther, Electrical and Computer Engineering Department, Utah State University, Logan, Utah. Authors: Authors: (1) Todd K. Moon, Electrical and Computer Engineering Department, Utah State University, Logan, Utah; (2) Jacob H. Gunther, Electrical and Computer Engineering Department, Utah State University, Logan, Utah. Table of Links Abstract and 1 Introduction and Background Abstract and 1 Introduction and Background 2 Statistical Parsing and Extracted Features 2 Statistical Parsing and Extracted Features 3 Parse Tree Features 3 Parse Tree Features 4 Classifier 4 Classifier 5 Dimension Reduction 5 Dimension Reduction 6 The Federalist Papers 6 The Federalist Papers 6.1 Sanditon 6.1 Sanditon 7 Conclusions, Discussion, and Future Work 7 Conclusions, Discussion, and Future Work A. A Brief Introduction to Statistical Parsing A. A Brief Introduction to Statistical Parsing B. Dimension Reduction: Some Mathematical Details B. Dimension Reduction: Some Mathematical Details References References Abstract Over the years there has been ongoing interest in detecting authorship of a text based on statistical properties of the text, such as by using occurrence rates of noncontextual words. In previous work, these techniques have been used, for example, to determine authorship of all of The Federalist Papers. Such methods may be useful in more modern times to detect fake or AI authorship. Progress in statistical natural language parsers introduces the possibility of using grammatical structure to detect authorship. In this paper we explore a new possibility for detecting authorship using grammatical structural information extracted using a statistical natural language parser. This paper provides a proof of concept, testing author classification based on grammatical structure on a set of “proof texts,” The Federalist Papers and Sanditon which have been as test cases in previous authorship detection studies. Several features extracted from the statistical natural language parser were explored: all subtrees of some depth from any level; rooted subtrees of some depth, part of speech, and part of speech by level in the parse tree. It was found to be helpful to project the features into a lower dimensional space. Statistical experiments on these documents demonstrate that information from a statistical parser can, in fact, assist in distinguishing authors. 1 Introduction and Background There has been considerable effort over the years related to using statistical methods to identify authorship of texts, based on examples from candidate authors, in what is sometimes called “stylometry” or “author identification.” Statistical analysis of documents goes back to Augustus de Morgan in 1851 [1, p. 282], [2, p. 166], who proposed that word length statistics might be used to determine the authorship of the Pauline epistles. Stylometry was employed as early as 1901 to explore the authorship of Shakespeare [3]. Since then, it has been employed in a variety of literary studies (see, e.g. [4, 5, 6]), including twelve of The Federalist Papers which were of uncertain authorship [7] — which we re-examine here — and an unfinished novel by Jane Austen —which we also re-examine here. Information theoretic techniques have also been used more recently [8]. Earlier work in stylometry has been based on “noncontextual words,” words which do not convey the primary meaning of the text, but which act in the background of the text to provide structure and flow. Noncontextual words are at least plausible, since an author may address a variety of topics, so particular distinguishing words are not necessarily revealing of authorship. In noncontextual word studies, a set of most common words noncontextual is selected [2], and documents are represented by word counts, or ratios of word counts to document length. A review of the statistical methods is in [9]. As a variation, sets of ratios of counts of noncontextual word patterns to other word patterns are also employed [10]. Statistical analysis based on author vocabulary size vs. document length — the “vocabulary richness” — has also been explored [11]. For other related work, see [12, 13, 14, 15] A more recent paper [16] considers the effectiveness of a wide variety of feature sets. Feature sets considered there include: vectors comprising frequencies of pronouns; function words (that is, articles, pronouns, particles, expletives); part of speech (POS); most common words; syntactic features (such as noun phrase, or verb phrase); or tense (e.g. use of present or past tense); voice (active of passive). In [16], feature vectors are formed from combinations of histograms, then reduced in dimensionality using a two-stage process of principle component analysis [17] followed by dimension reduction using linear discriminant analysis (LDA). In their LDA, the within-cluster scatter matrix is singular (due to the high dimension of the feature vectors relative the number of available training vectors), so their scatter matrix is regularized. To test this, the authors consider a range of regularization parameters, selecting one which gives the best performance. More recent work [18] mentions the survey in [15] in which commonly used features in the authorship field are word and character n-grams. As noted, there are risks the statistical methods might be biased by topic-related patterns. As [18] observe, “an authorship classifier (even a seemingly good one) might end up unintentionally performing topic identification if domain-dependent features are used. ... In order to avoid this, researchers might limit their scope to features that are clearly topic-agnostic, such as function words or syntactic features.” The work presented here falls in the latter category, making use of grammatical structures statistically extracted from the text. These appear to be difficult to spoof. Examination of other recent works [19, 20] indicate that there is ongoing interest in author identification methods, but none making use of the grammatical structures use here; there is a tendency to rely more on traditional n-grams. In this work the feature vectors are obtained using tree information from parse trees from a natural language parsing tool [21]. These features were not among the features considered in [16]. The grammatical structures are, it seems, more subtle than simple counts of classes of words, and hence may be less subject to spoofing or topic bias, since it seems unlikely that an author intending to imitate another would be able to coherently track complicated patterns of usage, and the features do not include any words from the documents. It is found that the tree-based features perform better than the POS features on the test data considered. The feature vectors so obtained can be of very high dimension, so dimension reduction is also performed here. However, to deal with the singularity of the within-cluster scatter matrix, a generalized SVD approach is used, which avoids the need to select a regularization parameter. This paper provides a proof-of-concept of these tree-based features to distinguish authorship by applying them to documents which have been previously examined, The Federalist Papers and Sanditon. The ability to classify by authorship is explored for several feature vectors obtained from the parsed information. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Your Writing Has a Fingerprint—And This Cutting Edge AI Model Can Identify It

Din skrift har et fingeraftryk - og denne banebrydende AI-model kan se det

Tu escritura tiene una huella digital y este modelo de inteligencia artificial de vanguardia puede detectarla

לכתיבה שלך יש טביעת אצבע - ודגם הבינה המלאכותית החדיש הזה יכול לזהות אותה

あなたの筆跡には指紋がある。最先端の AI モデルがそれを検出できる

Сиздин жазууңузда манжа изи бар жана бул эң алдыңкы AI модели аны байкай алат

Kokoma na yo Ezali na empreinte ya mosapi —Mpe Modèle Oyo ya AI ya sika Ekoki Komona Yango

ການຂຽນຂອງເຈົ້າມີລາຍນິ້ວມື - ແລະຕົວແບບ AI ທີ່ມີການຕັດແຂບນີ້ສາມາດເຫັນໄດ້

Tulisan Anda Mempunyai Cap Jari—Dan Model AI Cutting Edge Ini Dapat Mengesannya

Sua escrita tem uma impressão digital — e este modelo de IA de ponta pode identificá-la

การเขียนของคุณมีลายนิ้วมือ และโมเดล AI ล้ำสมัยนี้สามารถระบุลายนิ้วมือของคุณได้

Matsalwa Ya Wena Ma Ni Xikombiso Xa Tintiho—Naswona Modelo Lowu Wa AI Wa Mahungu Lamanene Wu Nga Swi Vona

Ubhalo lwakho lunophawu lweminwe-kwaye lo mzekelo wokusika we-AI unokuwubona

Authors:
(1) Diwen Xue, University of Michigan;
(2) Reethika Ramesh, University of Michigan;
(3) Arham Jain, University of Michigan;
(4) Arham Jain, Merit Network, Inc.;
(5) J. Alex Halderman, University of Michigan;
(6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad;
(7) Roya Ensaf, University of Michigan. Table of Links Abstract and 1 Introduction 2 Background & Related Work 3 Challenges in Real-world VPN Detection 4 Adversary Model and Deployment 5 Ethics, Privacy, and Responsible Disclosure 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6.2 ACK-based Fingerprinting 6.3 Active Server Fingerprinting 6.4 Constructing Filters and Probers 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7.2 Choice of Observation Window N 7.3 Effects of Packet Loss 7.4 Server Churn for Asynchronous Probing 7.5 Probe UDP and Obfuscated OpenVPN Servers 8 Real-world Deployment Setup 9 Evaluation & Findings and 9.1 Results for control VPN flows 9.2 Results for all flows 10 Discussion and Mitigations 11 Conclusion 12 Acknowledgement and References Appendix A Appendix This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Diwen Xue, University of Michigan; (2) Reethika Ramesh, University of Michigan; (3) Arham Jain, University of Michigan; (4) Arham Jain, Merit Network, Inc.; (5) J. Alex Halderman, University of Michigan; (6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad; (7) Roya Ensaf, University of Michigan. Authors: Authors: (1) Diwen Xue, University of Michigan; (2) Reethika Ramesh, University of Michigan; (3) Arham Jain, University of Michigan; (4) Arham Jain, Merit Network, Inc.; (5) J. Alex Halderman, University of Michigan; (6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad; (7) Roya Ensaf, University of Michigan. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Background & Related Work 2 Background & Related Work 3 Challenges in Real-world VPN Detection 3 Challenges in Real-world VPN Detection 4 Adversary Model and Deployment 4 Adversary Model and Deployment 5 Ethics, Privacy, and Responsible Disclosure 5 Ethics, Privacy, and Responsible Disclosure 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6.2 ACK-based Fingerprinting 6.2 ACK-based Fingerprinting 6.3 Active Server Fingerprinting 6.3 Active Server Fingerprinting 6.4 Constructing Filters and Probers 6.4 Constructing Filters and Probers 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7.2 Choice of Observation Window N 7.2 Choice of Observation Window N 7.3 Effects of Packet Loss 7.3 Effects of Packet Loss 7.4 Server Churn for Asynchronous Probing 7.4 Server Churn for Asynchronous Probing 7.5 Probe UDP and Obfuscated OpenVPN Servers 7.5 Probe UDP and Obfuscated OpenVPN Servers 8 Real-world Deployment Setup 8 Real-world Deployment Setup 9 Evaluation & Findings and 9.1 Results for control VPN flows 9 Evaluation & Findings and 9.1 Results for control VPN flows 9.2 Results for all flows 9.2 Results for all flows 10 Discussion and Mitigations 10 Discussion and Mitigations 11 Conclusion 11 Conclusion 12 Acknowledgement and References 12 Acknowledgement and References Appendix Appendix A Appendix This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

OpenVPN is Open to VPN Fingerprinting: Appendix

OpenVPN is Open to VPN Fingerprinting: Acknowledgement and References

Authors:
(1) Diwen Xue, University of Michigan;
(2) Reethika Ramesh, University of Michigan;
(3) Arham Jain, University of Michigan;
(4) Arham Jain, Merit Network, Inc.;
(5) J. Alex Halderman, University of Michigan;
(6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad;
(7) Roya Ensaf, University of Michigan. Table of Links Abstract and 1 Introduction 2 Background & Related Work 3 Challenges in Real-world VPN Detection 4 Adversary Model and Deployment 5 Ethics, Privacy, and Responsible Disclosure 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6.2 ACK-based Fingerprinting 6.3 Active Server Fingerprinting 6.4 Constructing Filters and Probers 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7.2 Choice of Observation Window N 7.3 Effects of Packet Loss 7.4 Server Churn for Asynchronous Probing 7.5 Probe UDP and Obfuscated OpenVPN Servers 8 Real-world Deployment Setup 9 Evaluation & Findings and 9.1 Results for control VPN flows 9.2 Results for all flows 10 Discussion and Mitigations 11 Conclusion 12 Acknowledgement and References Appendix 10 Discussion and Mitigations ISPs and government censors are motivated to detect OpenVPN flows in order to enforce traffic policies and information controls. We demonstrate that tracking and blocking the use of OpenVPN, even with most deployed obfuscation methods, is practical at scale and with minimal collateral damage. We note that many VPN providers’ claims that their obfuscated services are unobservable appear to be misleading and potentially dangerous, especially to users from countries where personal VPN usage is illegal. In light of our findings, users should not expect complete unobservability even when connected to “obfuscated” OpenVPN-based services. Putting the human danger aside, the ease of fingerprinting makes OpenVPN more susceptible to throttling or blocking from ISPs and governments. Previous research suggests that some censors already use two-stage pipelines, which are highly similar to our deployment, to detect other protocols such as Tor or Shadowsocks [1, 71]. These adversaries can quickly adapt such infrastructure to detect OpenVPN traffic by simply adding protocol-specific fingerprints and probes. Furthermore, while we focus on OpenVPN due to its overwhelming popularity among commercial VPNs, it is possible to extend our two-stage framework to other VPN protocols (e.g., WireGuard and StrongSwan) by analyzing their communication patterns and server behaviors. Governments can also quickly adopt these fingerprints to track and block VPN usage during sensitive times, like political upheavals, when VPN connections are most vital to the free flow of information. Short-Term Mitigation There are several defensive strategies to achieve near-term protection from the fingerprinting attack we describe. First, VPN providers offering both vanilla and obfuscated OpenVPN services should avoid colocating them. Ideally, obfuscation servers should be well separated from OpenVPN instances in the network address space and operate as “bridge servers” that forward client traffic to VPN servers elsewhere. For example, Mullvad VPN offers a Shadowsocks-based obfuscator service as a dedicated bridge, separating the VPN servers from the obfuscation [28]. Second, VPN providers should switch from static to random padding for their obfuscated services. As we have shown, for protocols with a stable and distinctive handshake phase, even the most basic threshold-based detector is able to fingerprint them by packet sizes. Ideally, the obfuscation layer should be able to send zero-length packets (packets whose payloads are all padding) to break the one-to-one correspondence between the unobfuscated and obfuscated packet streams [72]. Yet it is worth noting that previous work has shown that even fully randomized obfuscators (e.g., obfs4) can themselves be vulnerable to entropy-based fingerprinting attacks [67]. Third, we suggest that the OpenVPN developers follow recommendations from previous work with regard to how servers respond to failed handshake attempts. Servers closing failed connections immediately or in a predictable manner has enabled active probing attacks against a variety of other protocols [1, 13, 58]. In response, these protocols have implemented either unlimited timeouts (reading from the buffer indefinitely) or diversified close behaviors (in which each server instance closes failed connections in a different manner). Long-Term Defenses In the long term, we fear that the catand-mouse game between censors and circumvention tools, such as the Great Firewall and Tor, will occur in the VPN ecosystem as well, and developers and providers will have to adapt their obfuscation strategies to the evolving adversaries. We urge commercial VPN providers to adopt more standardized obfuscation solutions, such as Pluggable Transports [39], and to be more transparent about the techniques used by their obfuscated services. This transparency will help foster development of stronger obfuscation methods and encourage developers to design better techniques to overcome the progress of information control technologies. Additional future work is needed to characterize the performance costs of different approaches to VPN obfuscation and to help users with varying threat models make appropriate trade-offs between performance and resilient unobservability. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Diwen Xue, University of Michigan; (2) Reethika Ramesh, University of Michigan; (3) Arham Jain, University of Michigan; (4) Arham Jain, Merit Network, Inc.; (5) J. Alex Halderman, University of Michigan; (6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad; (7) Roya Ensaf, University of Michigan. Authors: Authors: (1) Diwen Xue, University of Michigan; (2) Reethika Ramesh, University of Michigan; (3) Arham Jain, University of Michigan; (4) Arham Jain, Merit Network, Inc.; (5) J. Alex Halderman, University of Michigan; (6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad; (7) Roya Ensaf, University of Michigan. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Background & Related Work 2 Background & Related Work 3 Challenges in Real-world VPN Detection 3 Challenges in Real-world VPN Detection 4 Adversary Model and Deployment 4 Adversary Model and Deployment 5 Ethics, Privacy, and Responsible Disclosure 5 Ethics, Privacy, and Responsible Disclosure 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6.2 ACK-based Fingerprinting 6.2 ACK-based Fingerprinting 6.3 Active Server Fingerprinting 6.3 Active Server Fingerprinting 6.4 Constructing Filters and Probers 6.4 Constructing Filters and Probers 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7.2 Choice of Observation Window N 7.2 Choice of Observation Window N 7.3 Effects of Packet Loss 7.3 Effects of Packet Loss 7.4 Server Churn for Asynchronous Probing 7.4 Server Churn for Asynchronous Probing 7.5 Probe UDP and Obfuscated OpenVPN Servers 7.5 Probe UDP and Obfuscated OpenVPN Servers 8 Real-world Deployment Setup 8 Real-world Deployment Setup 9 Evaluation & Findings and 9.1 Results for control VPN flows 9 Evaluation & Findings and 9.1 Results for control VPN flows 9.2 Results for all flows 9.2 Results for all flows 10 Discussion and Mitigations 10 Discussion and Mitigations 11 Conclusion 11 Conclusion 12 Acknowledgement and References 12 Acknowledgement and References Appendix Appendix 10 Discussion and Mitigations ISPs and government censors are motivated to detect OpenVPN flows in order to enforce traffic policies and information controls. We demonstrate that tracking and blocking the use of OpenVPN, even with most deployed obfuscation methods, is practical at scale and with minimal collateral damage. We note that many VPN providers’ claims that their obfuscated services are unobservable appear to be misleading and potentially dangerous, especially to users from countries where personal VPN usage is illegal. In light of our findings, users should not expect complete unobservability even when connected to “obfuscated” OpenVPN-based services. Putting the human danger aside, the ease of fingerprinting makes OpenVPN more susceptible to throttling or blocking from ISPs and governments. Previous research suggests that some censors already use two-stage pipelines, which are highly similar to our deployment, to detect other protocols such as Tor or Shadowsocks [1, 71]. These adversaries can quickly adapt such infrastructure to detect OpenVPN traffic by simply adding protocol-specific fingerprints and probes. Furthermore, while we focus on OpenVPN due to its overwhelming popularity among commercial VPNs, it is possible to extend our two-stage framework to other VPN protocols (e.g., WireGuard and StrongSwan) by analyzing their communication patterns and server behaviors. Governments can also quickly adopt these fingerprints to track and block VPN usage during sensitive times, like political upheavals, when VPN connections are most vital to the free flow of information. Short-Term Mitigation There are several defensive strategies to achieve near-term protection from the fingerprinting attack we describe. First, VPN providers offering both vanilla and obfuscated OpenVPN services should avoid colocating them. Ideally, obfuscation servers should be well separated from OpenVPN instances in the network address space and operate as “bridge servers” that forward client traffic to VPN servers elsewhere. For example, Mullvad VPN offers a Shadowsocks-based obfuscator service as a dedicated bridge, separating the VPN servers from the obfuscation [28]. Short-Term Mitigation Second, VPN providers should switch from static to random padding for their obfuscated services. As we have shown, for protocols with a stable and distinctive handshake phase, even the most basic threshold-based detector is able to fingerprint them by packet sizes. Ideally, the obfuscation layer should be able to send zero-length packets (packets whose payloads are all padding) to break the one-to-one correspondence between the unobfuscated and obfuscated packet streams [72]. Yet it is worth noting that previous work has shown that even fully randomized obfuscators (e.g., obfs4) can themselves be vulnerable to entropy-based fingerprinting attacks [67]. Third, we suggest that the OpenVPN developers follow recommendations from previous work with regard to how servers respond to failed handshake attempts. Servers closing failed connections immediately or in a predictable manner has enabled active probing attacks against a variety of other protocols [1, 13, 58]. In response, these protocols have implemented either unlimited timeouts (reading from the buffer indefinitely) or diversified close behaviors (in which each server instance closes failed connections in a different manner). Long-Term Defenses In the long term, we fear that the catand-mouse game between censors and circumvention tools, such as the Great Firewall and Tor, will occur in the VPN ecosystem as well, and developers and providers will have to adapt their obfuscation strategies to the evolving adversaries. We urge commercial VPN providers to adopt more standardized obfuscation solutions, such as Pluggable Transports [39], and to be more transparent about the techniques used by their obfuscated services. This transparency will help foster development of stronger obfuscation methods and encourage developers to design better techniques to overcome the progress of information control technologies. Additional future work is needed to characterize the performance costs of different approaches to VPN obfuscation and to help users with varying threat models make appropriate trade-offs between performance and resilient unobservability. Long-Term Defenses This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Mitigating OpenVPN Fingerprinting: Risks and Defensive Strategies

Authors:
(1) Diwen Xue, University of Michigan;
(2) Reethika Ramesh, University of Michigan;
(3) Arham Jain, University of Michigan;
(4) Arham Jain, Merit Network, Inc.;
(5) J. Alex Halderman, University of Michigan;
(6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad;
(7) Roya Ensaf, University of Michigan. Table of Links Abstract and 1 Introduction 2 Background & Related Work 3 Challenges in Real-world VPN Detection 4 Adversary Model and Deployment 5 Ethics, Privacy, and Responsible Disclosure 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6.2 ACK-based Fingerprinting 6.3 Active Server Fingerprinting 6.4 Constructing Filters and Probers 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7.2 Choice of Observation Window N 7.3 Effects of Packet Loss 7.4 Server Churn for Asynchronous Probing 7.5 Probe UDP and Obfuscated OpenVPN Servers 8 Real-world Deployment Setup 9 Evaluation & Findings and 9.1 Results for control VPN flows 9.2 Results for all flows 10 Discussion and Mitigations 11 Conclusion 12 Acknowledgement and References Appendix 8 Real-world Deployment Setup We set out to explore if an ISP or censor can fingerprint OpenVPN connections at scale, without significant collateral damage. Adopting the viewpoint of an adversarial ISP, we deploy our framework inside Merit, as shown in Figure 2. Our evaluation is two-fold: we generate control vanilla and obfuscated flows with commercial VPN providers and attempt to identify them as a network intermediary; we also process other traffic passing through our Monitoring Station in order to estimate the false positive rate of our framework. We set up our framework on a 16-core server (Monitoring Station) inside Merit with two mirroring interfaces that have an aggregated 20 Gbps bandwidth. Due to the large traffic volume, we optimize our deployment with PF_RING [38] in order to improve the packet processing speed. We employ PF_- RING in zero-copy mode and spread the traffic load across a Zeek cluster of 15 workers. Nonetheless, due to limited CPU resources, we only sample 12.5% of all TCP and UDP flows arriving at the network interfaces in order to minimize the effect of packet loss. The sampling is based on IP pairs so that all bi-directional traffic of a flow will be selected/dropped together. With these settings, we are able to operate with an end-to-end packet loss rate under 3%. Even though we process only a fraction of all traffic, our Filter still handles over 15 Terabytes of traffic from over 2 billion flows on an average day on a single server. In addition, processing all traffic without sampling is feasible through parallelism or using faster CPUs. Next, we set up Probers on two dedicated measurement machines, each provisioned with 10 IPv4 and 1 IPv6 addresses. By the end of each day during the evaluation, the Probers fetch filtering logs from the Monitoring Station. For each target, we run a Masscan [25] to the /29 subnet the IP belongs to over all TCP ports (1-65535). We follow up each discovered open port by running our probing scheme, and endpoints confirmed through probing are recorded for manual analysis. To select VPN services for evaluation, we first generate a list of “top” VPN services ranked by popularity. We combine 80 providers, most of which are paid premium VPN services, from top VPN recommendation sites based on previous work [42], listed in Appendix Table 4. Next, we visit the websites of these VPN providers searching for “Obfuscation”, “Stealth”, or “Camouflage Mode” etc., and include providers that offer at least one obfuscated VPN configuration. In total, we find 24 providers offering obfuscated services. We test all obfuscation configurations if more than one is offered as well as vanilla OpenVPN for each provider. If TCP and UDP modes are both available, we test them separately. In total, we have 81 configurations, 41 of which are obfuscated ones. We configure the Client Station inside Merit to act as a VPN client. Both upstream and downstream traffic of the Client Station go through the router that mirrors traffic to the Monitoring Station. In addition, we exclude this server from our random sampling so that all traffic to/from this server will be analyzed. On the client, we run an automated script to generate control traffic for our evaluation. For each iteration, we start the VPN client application and connect to the “default / recommended” server using Pywinauto [41]. After a random wait of 20 to 180 seconds, we confirm that the VPN tunnel is active and generate random browsing traffic with Selenium [45] by sending requests to a random website from the Alexa top 500. Finally, we disconnect from the VPN server and wait for 180 seconds before proceeding to the next iteration. For each VPN configuration, we repeat the process 50 times and collect packet captures for reference. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Diwen Xue, University of Michigan; (2) Reethika Ramesh, University of Michigan; (3) Arham Jain, University of Michigan; (4) Arham Jain, Merit Network, Inc.; (5) J. Alex Halderman, University of Michigan; (6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad; (7) Roya Ensaf, University of Michigan. Authors: Authors: (1) Diwen Xue, University of Michigan; (2) Reethika Ramesh, University of Michigan; (3) Arham Jain, University of Michigan; (4) Arham Jain, Merit Network, Inc.; (5) J. Alex Halderman, University of Michigan; (6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad; (7) Roya Ensaf, University of Michigan. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Background & Related Work 2 Background & Related Work 3 Challenges in Real-world VPN Detection 3 Challenges in Real-world VPN Detection 4 Adversary Model and Deployment 4 Adversary Model and Deployment 5 Ethics, Privacy, and Responsible Disclosure 5 Ethics, Privacy, and Responsible Disclosure 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6.2 ACK-based Fingerprinting 6.2 ACK-based Fingerprinting 6.3 Active Server Fingerprinting 6.3 Active Server Fingerprinting 6.4 Constructing Filters and Probers 6.4 Constructing Filters and Probers 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7.2 Choice of Observation Window N 7.2 Choice of Observation Window N 7.3 Effects of Packet Loss 7.3 Effects of Packet Loss 7.4 Server Churn for Asynchronous Probing 7.4 Server Churn for Asynchronous Probing 7.5 Probe UDP and Obfuscated OpenVPN Servers 7.5 Probe UDP and Obfuscated OpenVPN Servers 8 Real-world Deployment Setup 8 Real-world Deployment Setup 9 Evaluation & Findings and 9.1 Results for control VPN flows 9 Evaluation & Findings and 9.1 Results for control VPN flows 9.2 Results for all flows 9.2 Results for all flows 10 Discussion and Mitigations 10 Discussion and Mitigations 11 Conclusion 11 Conclusion 12 Acknowledgement and References 12 Acknowledgement and References Appendix Appendix 8 Real-world Deployment Setup We set out to explore if an ISP or censor can fingerprint OpenVPN connections at scale, without significant collateral damage. Adopting the viewpoint of an adversarial ISP, we deploy our framework inside Merit, as shown in Figure 2. Our evaluation is two-fold: we generate control vanilla and obfuscated flows with commercial VPN providers and attempt to identify them as a network intermediary; we also process other traffic passing through our Monitoring Station in order to estimate the false positive rate of our framework. We set up our framework on a 16-core server (Monitoring Station) inside Merit with two mirroring interfaces that have an aggregated 20 Gbps bandwidth. Due to the large traffic volume, we optimize our deployment with PF_RING [38] in order to improve the packet processing speed. We employ PF_- RING in zero-copy mode and spread the traffic load across a Zeek cluster of 15 workers. Nonetheless, due to limited CPU resources, we only sample 12.5% of all TCP and UDP flows arriving at the network interfaces in order to minimize the effect of packet loss. The sampling is based on IP pairs so that all bi-directional traffic of a flow will be selected/dropped together. With these settings, we are able to operate with an end-to-end packet loss rate under 3%. Even though we process only a fraction of all traffic, our Filter still handles over 15 Terabytes of traffic from over 2 billion flows on an average day on a single server. In addition, processing all traffic without sampling is feasible through parallelism or using faster CPUs. Next, we set up Probers on two dedicated measurement machines, each provisioned with 10 IPv4 and 1 IPv6 addresses. By the end of each day during the evaluation, the Probers fetch filtering logs from the Monitoring Station. For each target, we run a Masscan [25] to the /29 subnet the IP belongs to over all TCP ports (1-65535). We follow up each discovered open port by running our probing scheme, and endpoints confirmed through probing are recorded for manual analysis. To select VPN services for evaluation, we first generate a list of “top” VPN services ranked by popularity. We combine 80 providers, most of which are paid premium VPN services, from top VPN recommendation sites based on previous work [42], listed in Appendix Table 4. Next, we visit the websites of these VPN providers searching for “Obfuscation”, “Stealth”, or “Camouflage Mode” etc., and include providers that offer at least one obfuscated VPN configuration. In total, we find 24 providers offering obfuscated services. We test all obfuscation configurations if more than one is offered as well as vanilla OpenVPN for each provider. If TCP and UDP modes are both available, we test them separately. In total, we have 81 configurations, 41 of which are obfuscated ones. We configure the Client Station inside Merit to act as a VPN client. Both upstream and downstream traffic of the Client Station go through the router that mirrors traffic to the Monitoring Station. In addition, we exclude this server from our random sampling so that all traffic to/from this server will be analyzed. On the client, we run an automated script to generate control traffic for our evaluation. For each iteration, we start the VPN client application and connect to the “default / recommended” server using Pywinauto [41]. After a random wait of 20 to 180 seconds, we confirm that the VPN tunnel is active and generate random browsing traffic with Selenium [45] by sending requests to a random website from the Alexa top 500. Finally, we disconnect from the VPN server and wait for 180 seconds before proceeding to the next iteration. For each VPN configuration, we repeat the process 50 times and collect packet captures for reference. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

OpenVPN is Open to VPN Fingerprinting: Real-world Deployment Setup

Authors:
(1) Diwen Xue, University of Michigan;
(2) Reethika Ramesh, University of Michigan;
(3) Arham Jain, University of Michigan;
(4) Arham Jain, Merit Network, Inc.;
(5) J. Alex Halderman, University of Michigan;
(6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad;
(7) Roya Ensaf, University of Michigan. Table of Links Abstract and 1 Introduction 2 Background & Related Work 3 Challenges in Real-world VPN Detection 4 Adversary Model and Deployment 5 Ethics, Privacy, and Responsible Disclosure 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6.2 ACK-based Fingerprinting 6.3 Active Server Fingerprinting 6.4 Constructing Filters and Probers 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7.2 Choice of Observation Window N 7.3 Effects of Packet Loss 7.4 Server Churn for Asynchronous Probing 7.5 Probe UDP and Obfuscated OpenVPN Servers 8 Real-world Deployment Setup 9 Evaluation & Findings and 9.1 Results for control VPN flows 9.2 Results for all flows 10 Discussion and Mitigations 11 Conclusion 12 Acknowledgement and References Appendix 6.4 Constructing Filters and Probers Our Filter performs both opcode and ACK-based fingerprinting, flagging a flow if at least one fingerprint matches. This is because the opcode and ACK fingerprints are designed to be complementary: both are effective against vanilla OpenVPN and they each target a specific subset of obfuscations. The former works against XOR-based obfuscations that work like Vigenère ciphers, i.e. they always encrypt the same plaintext opcodes at the same position to the same ciphertext bytes. The latter targets tunneling-based obfuscation that lacks random padding and preserves the 1:1 correspondence between the original and obfuscated packet streams. Combining the two features maximizes our fingerprinting coverage, as we discovered that even within the same provider, obfuscating strategies can vary a lot (§ 9). Table 5 in Appendix shows the effectiveness of each feature against each commercial VPN service we tested. Following Filter’s result, the Prober performs the active probing scheme to further lower potential false positives. We implement the Filter in Zeek [75], an open-source network monitoring tool. We note that the evaluation processes for opcode and ACK-based fingerprinting are quite simple: both only require several dozen integer comparisons (limited by the observation window) while maintaining a small number of per-flow states. We implement the Prober in Nim [31]. We believe that both components can be easily deployed by any ISP or censor. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Diwen Xue, University of Michigan; (2) Reethika Ramesh, University of Michigan; (3) Arham Jain, University of Michigan; (4) Arham Jain, Merit Network, Inc.; (5) J. Alex Halderman, University of Michigan; (6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad; (7) Roya Ensaf, University of Michigan. Authors: Authors: (1) Diwen Xue, University of Michigan; (2) Reethika Ramesh, University of Michigan; (3) Arham Jain, University of Michigan; (4) Arham Jain, Merit Network, Inc.; (5) J. Alex Halderman, University of Michigan; (6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad; (7) Roya Ensaf, University of Michigan. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Background & Related Work 2 Background & Related Work 3 Challenges in Real-world VPN Detection 3 Challenges in Real-world VPN Detection 4 Adversary Model and Deployment 4 Adversary Model and Deployment 5 Ethics, Privacy, and Responsible Disclosure 5 Ethics, Privacy, and Responsible Disclosure 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6.2 ACK-based Fingerprinting 6.2 ACK-based Fingerprinting 6.3 Active Server Fingerprinting 6.3 Active Server Fingerprinting 6.4 Constructing Filters and Probers 6.4 Constructing Filters and Probers 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7.2 Choice of Observation Window N 7.2 Choice of Observation Window N 7.3 Effects of Packet Loss 7.3 Effects of Packet Loss 7.4 Server Churn for Asynchronous Probing 7.4 Server Churn for Asynchronous Probing 7.5 Probe UDP and Obfuscated OpenVPN Servers 7.5 Probe UDP and Obfuscated OpenVPN Servers 8 Real-world Deployment Setup 8 Real-world Deployment Setup 9 Evaluation & Findings and 9.1 Results for control VPN flows 9 Evaluation & Findings and 9.1 Results for control VPN flows 9.2 Results for all flows 9.2 Results for all flows 10 Discussion and Mitigations 10 Discussion and Mitigations 11 Conclusion 11 Conclusion 12 Acknowledgement and References 12 Acknowledgement and References Appendix Appendix 6.4 Constructing Filters and Probers Filters Our Filter performs both opcode and ACK-based fingerprinting, flagging a flow if at least one fingerprint matches. This is because the opcode and ACK fingerprints are designed to be complementary: both are effective against vanilla OpenVPN and they each target a specific subset of obfuscations. The former works against XOR-based obfuscations that work like Vigenère ciphers, i.e. they always encrypt the same plaintext opcodes at the same position to the same ciphertext bytes. The latter targets tunneling-based obfuscation that lacks random padding and preserves the 1:1 correspondence between the original and obfuscated packet streams. Combining the two features maximizes our fingerprinting coverage, as we discovered that even within the same provider, obfuscating strategies can vary a lot (§ 9). Table 5 in Appendix shows the effectiveness of each feature against each commercial VPN service we tested. Following Filter’s result, the Prober performs the active probing scheme to further lower potential false positives. Filter Filter’s Prober We implement the Filter in Zeek [75], an open-source network monitoring tool. We note that the evaluation processes for opcode and ACK-based fingerprinting are quite simple: both only require several dozen integer comparisons (limited by the observation window) while maintaining a small number of per-flow states. We implement the Prober in Nim [31]. We believe that both components can be easily deployed by any ISP or censor. Filter Prober This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Enhancing OpenVPN Detection: Combining Fingerprinting and Active Probing

Authors:
(1) Diwen Xue, University of Michigan;
(2) Reethika Ramesh, University of Michigan;
(3) Arham Jain, University of Michigan;
(4) Arham Jain, Merit Network, Inc.;
(5) J. Alex Halderman, University of Michigan;
(6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad;
(7) Roya Ensaf, University of Michigan. Table of Links Abstract and 1 Introduction 2 Background & Related Work 3 Challenges in Real-world VPN Detection 4 Adversary Model and Deployment 5 Ethics, Privacy, and Responsible Disclosure 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6.2 ACK-based Fingerprinting 6.3 Active Server Fingerprinting 6.4 Constructing Filters and Probers 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7.2 Choice of Observation Window N 7.3 Effects of Packet Loss 7.4 Server Churn for Asynchronous Probing 7.5 Probe UDP and Obfuscated OpenVPN Servers 8 Real-world Deployment Setup 9 Evaluation & Findings and 9.1 Results for control VPN flows 9.2 Results for all flows 10 Discussion and Mitigations 11 Conclusion 12 Acknowledgement and References Appendix 6.3 Active Server Fingerprinting We explore the feasibility of identifying an OpenVPN server through active probing. Typically, OpenVPN servers respond to a client reset with an explicit server reset, thereby giving away their identity. However, most commercial providers now have adopted tls-auth or tls-crypt options [50]. These options add an additional HMAC signature—signed by a pre-shared key—to every control channel packet for integrity verification, including the initial reset packets. With either of these options enabled, an OpenVPN server would not respond to an unauthenticated client reset with a server reset, but would instead drop such packets without further processing. The presence of such HMAC mechanism increases the complexity of doing active probing: it effectively makes OpenVPN servers “probe-resistant” [13] by remaining silent when probed by an unauthenticated client. In fact, similar HMAC mechanisms are used by more popular “probe-resistant” proxies, such as obfs4 [33]. However, unlike obfs4 which waits for a server-specific random delay before dropping an unauthenticated connection, OpenVPN always immediately closes the connection if a valid HMAC cannot be located. We design our probes to leverage this protocol-specific behavior, and as a result, we manage to fingerprint OpenVPN servers even if they do not respond throughout our probing cycles. The key concept is that although the application may not respond to probing, an attacker may still be able to fingerprint application-specific thresholds at the TCP level, such as timeouts or RST thresholds, as demonstrated by Frolov et al. [13]. We use two datasets in this section to help with designing probes. ZMap Set: to construct a realistic non-VPN endpoints dataset, we use ZMap to scan each of the 65,535 TCP ports over the entire IPv4 space, limiting results for each port to 200 endpoints (with the specific port open), resulting in over 13 million endpoints. Censys Set: We query the Censys.io [10] database for hosts with TCP port 1194/OpenVPN open. Next, we probe each endpoint with a typical OpenVPN Client Reset and group endpoints that respond with explicit Server Resets. This results in 180,858 hosts known to be OpenVPN endpoints (with “tls-auth” disabled). 6.3.1 Base Probes We design probes exploiting a behavior associated with how OpenVPN packetizes TCP streams. When OpenVPN operates over TCP, it needs to split the continuous stream into discrete OpenVPN packets. Figure 5 presents a high-level abstraction of this process. The most relevant parts are: a buffer is allocated in memory to reassemble fragments of OpenVPN packets encapsulated in TCP streams. The length N for the next OpenVPN packet is extracted from the first two bytes of the header (see Figure 3), and the routine keeps reading N additional bytes before it returns the reassembled packet to the caller. This means that an OpenVPN packet will not be parsed and checked for syntax and encryption errors until all its parts arrive at the server. Based on this behavior, we design two sequential probes to trigger an OpenVPN server into different code paths—which result in different connection timeouts— and measure the time elapsed before the server responds or terminates the connection. As shown in Table 1, Base Probe 1 carries a typical 16-byte OpenVPN Client Reset, while Base Probe 2 has the same payload with the last byte stripped off. The assumption is since our two probes only differ in one byte, most non-OpenVPN servers will respond to our probes in a similar way. However, for an OpenVPN server with HMAC enabled, the connection sending the first probe will be dropped immediately because the OpenVPN packet is reassembled and a valid HMAC cannot be located. The second probe will not receive an immediate response, as the server will wait for an additional byte to arrive for reassembly. The connection will stay idle until a server specific handshake timeout has passed, after which the connection will be dropped. As such, the first probe will be dropped at the decryption routine, while the second probe will be dropped at the packet reassembly routine, both labeled red in Figure 5. 6.3.2 Additional Probes The two probes, although useful, are limited and there may be other protocols with behaviors similar to OpenVPN. After using both to probe the ZMap Set, we still identify a handful of services that respond similarly to OpenVPN servers, such as Microsoft WBT Server (3389), Microsoft Message Queuing (1801), and Erlang Port Mapper Daemon (4369). We design additional probes based on the fact that OpenVPN validates packet length and will drop connections sending invalid length without waiting for the next packet to be reassembled. Here, packet length refers to the length declared by the first two bytes of an OpenVPN header (see Figure 3), rather than the TCP packet length. A “valid” length is in the range of [1, max_len], where max_len is derived from the server’s MTU configurations. For instance, default TUN MTU of 1500 bytes, combined with overheads (crypto IV, packet length, etc.), results in a max_len of 1627 bytes. In this case, probes whose first two bytes have a decimal value greater than 1627 (0x06,0x5B) will be dropped immediately. We also design probes leveraging the way a Linux server closes a TCP connection. When a TCP connection terminates, the operating systems at both ends typically complete a FIN 4-way handshake. However, previous work has found that if a connection is closed with unread bytes in buffer, Linux will send a RST packet [13]. A server’s “RST Threshold” is defined as the minimum number of bytes needed to send to the server to trigger a RST. We determine the RST threshold distribution for both ZMap Set and Censys Set. As shown in Figure 6, the vast majority of OpenVPN servers have a RST threshold around 1550-1660 bytes, corresponding to buffers allocated with typical MTU configurations. In contrast, over 97% of random ZMap endpoints have a RST threshold less than 500 or greater than 4000. We therefore construct an additional probe with 2,000 random bytes, which we expect over 98% of legitimate OpenVPN servers and less than 3% of random servers to respond to with RST packets. It is worth noting that the majority of “typical” HTTP, TLS, and SSH servers have already been filtered out by our base probes, so endpoints that respond at this stage are likely sharing the port with another service, thus warranting manual analysis (e.g., checking TLS certificate). While these three services are what we commonly observed, there may be instances where other services are running along with OpenVPN. This could lead to false negatives. Table 1 lists all probes and the expected behaviors from an OpenVPN server. An evaluation process is shown in Appendix Figure 11. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Diwen Xue, University of Michigan; (2) Reethika Ramesh, University of Michigan; (3) Arham Jain, University of Michigan; (4) Arham Jain, Merit Network, Inc.; (5) J. Alex Halderman, University of Michigan; (6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad; (7) Roya Ensaf, University of Michigan. Authors: Authors: (1) Diwen Xue, University of Michigan; (2) Reethika Ramesh, University of Michigan; (3) Arham Jain, University of Michigan; (4) Arham Jain, Merit Network, Inc.; (5) J. Alex Halderman, University of Michigan; (6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad; (7) Roya Ensaf, University of Michigan. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Background & Related Work 2 Background & Related Work 3 Challenges in Real-world VPN Detection 3 Challenges in Real-world VPN Detection 4 Adversary Model and Deployment 4 Adversary Model and Deployment 5 Ethics, Privacy, and Responsible Disclosure 5 Ethics, Privacy, and Responsible Disclosure 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6.2 ACK-based Fingerprinting 6.2 ACK-based Fingerprinting 6.3 Active Server Fingerprinting 6.3 Active Server Fingerprinting 6.4 Constructing Filters and Probers 6.4 Constructing Filters and Probers 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7.2 Choice of Observation Window N 7.2 Choice of Observation Window N 7.3 Effects of Packet Loss 7.3 Effects of Packet Loss 7.4 Server Churn for Asynchronous Probing 7.4 Server Churn for Asynchronous Probing 7.5 Probe UDP and Obfuscated OpenVPN Servers 7.5 Probe UDP and Obfuscated OpenVPN Servers 8 Real-world Deployment Setup 8 Real-world Deployment Setup 9 Evaluation & Findings and 9.1 Results for control VPN flows 9 Evaluation & Findings and 9.1 Results for control VPN flows 9.2 Results for all flows 9.2 Results for all flows 10 Discussion and Mitigations 10 Discussion and Mitigations 11 Conclusion 11 Conclusion 12 Acknowledgement and References 12 Acknowledgement and References Appendix Appendix 6.3 Active Server Fingerprinting We explore the feasibility of identifying an OpenVPN server through active probing. Typically, OpenVPN servers respond to a client reset with an explicit server reset, thereby giving away their identity. However, most commercial providers now have adopted tls-auth or tls-crypt options [50]. These options add an additional HMAC signature—signed by a pre-shared key—to every control channel packet for integrity verification, including the initial reset packets. With either of these options enabled, an OpenVPN server would not respond to an unauthenticated client reset with a server reset, but would instead drop such packets without further processing. The presence of such HMAC mechanism increases the complexity of doing active probing: it effectively makes OpenVPN servers “probe-resistant” [13] by remaining silent when probed by an unauthenticated client. In fact, similar HMAC mechanisms are used by more popular “probe-resistant” proxies, such as obfs4 [33]. However, unlike obfs4 which waits for a server-specific random delay before dropping an unauthenticated connection, OpenVPN always immediately closes the connection if a valid HMAC cannot be located. We design our probes to leverage this protocol-specific behavior, and as a result, we manage to fingerprint OpenVPN servers even if they do not respond throughout our probing cycles. The key concept is that although the application may not respond to probing, an attacker may still be able to fingerprint application-specific thresholds at the TCP level, such as timeouts or RST thresholds, as demonstrated by Frolov et al. [13]. We use two datasets in this section to help with designing probes. ZMap Set: to construct a realistic non-VPN endpoints dataset, we use ZMap to scan each of the 65,535 TCP ports over the entire IPv4 space, limiting results for each port to 200 endpoints (with the specific port open), resulting in over 13 million endpoints. Censys Set: We query the Censys.io [10] database for hosts with TCP port 1194/OpenVPN open. Next, we probe each endpoint with a typical OpenVPN Client Reset and group endpoints that respond with explicit Server Resets. This results in 180,858 hosts known to be OpenVPN endpoints (with “tls-auth” disabled). 6.3.1 Base Probes 6.3.1 Base Probes We design probes exploiting a behavior associated with how OpenVPN packetizes TCP streams. When OpenVPN operates over TCP, it needs to split the continuous stream into discrete OpenVPN packets. Figure 5 presents a high-level abstraction of this process. The most relevant parts are: a buffer is allocated in memory to reassemble fragments of OpenVPN packets encapsulated in TCP streams. The length N for the next OpenVPN packet is extracted from the first two bytes of the header (see Figure 3), and the routine keeps reading N additional bytes before it returns the reassembled packet to the caller. This means that an OpenVPN packet will not be parsed and checked for syntax and encryption errors until all its parts arrive at the server. Based on this behavior, we design two sequential probes to trigger an OpenVPN server into different code paths—which result in different connection timeouts— and measure the time elapsed before the server responds or terminates the connection. As shown in Table 1, Base Probe 1 carries a typical 16-byte OpenVPN Client Reset, while Base Probe 2 has the same payload with the last byte stripped off. The assumption is since our two probes only differ in one byte, most non-OpenVPN servers will respond to our probes in a similar way. However, for an OpenVPN server with HMAC enabled, the connection sending the first probe will be dropped immediately because the OpenVPN packet is reassembled and a valid HMAC cannot be located. The second probe will not receive an immediate response, as the server will wait for an additional byte to arrive for reassembly. The connection will stay idle until a server specific handshake timeout has passed, after which the connection will be dropped. As such, the first probe will be dropped at the decryption routine, while the second probe will be dropped at the packet reassembly routine, both labeled red in Figure 5. 6.3.2 Additional Probes 6.3.2 Additional Probes The two probes, although useful, are limited and there may be other protocols with behaviors similar to OpenVPN. After using both to probe the ZMap Set, we still identify a handful of services that respond similarly to OpenVPN servers, such as Microsoft WBT Server (3389), Microsoft Message Queuing (1801), and Erlang Port Mapper Daemon (4369). We design additional probes based on the fact that OpenVPN validates packet length and will drop connections sending invalid length without waiting for the next packet to be reassembled. Here, packet length refers to the length declared by the first two bytes of an OpenVPN header (see Figure 3), rather than the TCP packet length. A “valid” length is in the range of [1, max_len], where max_len is derived from the server’s MTU configurations. For instance, default TUN MTU of 1500 bytes, combined with overheads (crypto IV, packet length, etc.), results in a max_len of 1627 bytes. In this case, probes whose first two bytes have a decimal value greater than 1627 (0x06,0x5B) will be dropped immediately. We also design probes leveraging the way a Linux server closes a TCP connection. When a TCP connection terminates, the operating systems at both ends typically complete a FIN 4-way handshake. However, previous work has found that if a connection is closed with unread bytes in buffer, Linux will send a RST packet [13]. A server’s “RST Threshold” is defined as the minimum number of bytes needed to send to the server to trigger a RST. We determine the RST threshold distribution for both ZMap Set and Censys Set. As shown in Figure 6, the vast majority of OpenVPN servers have a RST threshold around 1550-1660 bytes, corresponding to buffers allocated with typical MTU configurations. In contrast, over 97% of random ZMap endpoints have a RST threshold less than 500 or greater than 4000. We therefore construct an additional probe with 2,000 random bytes, which we expect over 98% of legitimate OpenVPN servers and less than 3% of random servers to respond to with RST packets. It is worth noting that the majority of “typical” HTTP, TLS, and SSH servers have already been filtered out by our base probes, so endpoints that respond at this stage are likely sharing the port with another service, thus warranting manual analysis (e.g., checking TLS certificate). While these three services are what we commonly observed, there may be instances where other services are running along with OpenVPN. This could lead to false negatives. Table 1 lists all probes and the expected behaviors from an OpenVPN server. An evaluation process is shown in Appendix Figure 11. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Active Probing: How OpenVPN Servers Are Fingerprinted Through Unique Behaviors

Authors:
(1) Diwen Xue, University of Michigan;
(2) Reethika Ramesh, University of Michigan;
(3) Arham Jain, University of Michigan;
(4) Arham Jain, Merit Network, Inc.;
(5) J. Alex Halderman, University of Michigan;
(6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad;
(7) Roya Ensaf, University of Michigan. Table of Links Abstract and 1 Introduction 2 Background & Related Work 3 Challenges in Real-world VPN Detection 4 Adversary Model and Deployment 5 Ethics, Privacy, and Responsible Disclosure 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6.2 ACK-based Fingerprinting 6.3 Active Server Fingerprinting 6.4 Constructing Filters and Probers 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7.2 Choice of Observation Window N 7.3 Effects of Packet Loss 7.4 Server Churn for Asynchronous Probing 7.5 Probe UDP and Obfuscated OpenVPN Servers 8 Real-world Deployment Setup 9 Evaluation & Findings and 9.1 Results for control VPN flows 9.2 Results for all flows 10 Discussion and Mitigations 11 Conclusion 12 Acknowledgement and References Appendix 6.2 ACK-based Fingerprinting OpenVPN engages in a TLS-style handshake with its peer over the control channel. Since TLS is designed to operate over a reliable layer, OpenVPN implements an explicit acknowledgement and re-transmission mechanism for its control channel messages [30]. Specifically, incoming P_- Control packets are acknowledged by P_ACK packets, which do not carry any TLS payloads and are uniform in size (Note these ACK packets are carried over by TCP as payload and are not the same as TCP ACK flags). Moreover, these ACK packets are seen mostly only in the early stage of a flow, during the handshake phase, and are not used in the actual data transfer channel, which can run over an unreliable layer. To our knowledge, we are the first to devise fingerprinting attempts based on the distinct protocol-layer ACKs against OpenVPN. Previously, the unique timing pattern in meek’s TCP-level ACK traffic has rendered the obfuscation tool vulnerable to detection [67]. For OpenVPN, the presence of explicit ACK packets, uniform in size and only seen in some parts of a session, provides another fingerprintable feature. Specifically, we first identify a likely ACK packet of a session by locating an initial packet exchange sequence of C- >S (Client-Reset), S->C (Server-Reset), C->S (ACK), C->S (Control), as illustrated in Figure 1. For vanilla OpenVPN and XOR-based obfuscation, the first ACK packet usually appears as the third (data) packet transmitted in a session. For tunnels or obfuscators that have their own handshake or key exchange process (e.g., Stunnel, SSH tunnel, or Obfsproxy), this counting is offset by the number of tunnel handshake packets. Next, we group packets into 10-packet bins, and we derive the ACK fingerprint for each flow by counting the number of packets in each bin that have the same size as the identified ACK packet. For OpenVPN flows, we expect to observe a high number of ACK packets in early bins and an absence of them in later bins. (Later in the session, Control and ACK packets can be exchanged again to transfer random key materials, but it is not expected to be observed within our observation window N.) This approach proves effective to fingerprint vanilla OpenVPN as well as obfuscated services running over encrypted tunnels that lack random padding. We quantify exact fingerprinting thresholds in Section 7.1. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Diwen Xue, University of Michigan; (2) Reethika Ramesh, University of Michigan; (3) Arham Jain, University of Michigan; (4) Arham Jain, Merit Network, Inc.; (5) J. Alex Halderman, University of Michigan; (6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad; (7) Roya Ensaf, University of Michigan. Authors: Authors: (1) Diwen Xue, University of Michigan; (2) Reethika Ramesh, University of Michigan; (3) Arham Jain, University of Michigan; (4) Arham Jain, Merit Network, Inc.; (5) J. Alex Halderman, University of Michigan; (6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad; (7) Roya Ensaf, University of Michigan. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Background & Related Work 2 Background & Related Work 3 Challenges in Real-world VPN Detection 3 Challenges in Real-world VPN Detection 4 Adversary Model and Deployment 4 Adversary Model and Deployment 5 Ethics, Privacy, and Responsible Disclosure 5 Ethics, Privacy, and Responsible Disclosure 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6.2 ACK-based Fingerprinting 6.2 ACK-based Fingerprinting 6.3 Active Server Fingerprinting 6.3 Active Server Fingerprinting 6.4 Constructing Filters and Probers 6.4 Constructing Filters and Probers 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7.2 Choice of Observation Window N 7.2 Choice of Observation Window N 7.3 Effects of Packet Loss 7.3 Effects of Packet Loss 7.4 Server Churn for Asynchronous Probing 7.4 Server Churn for Asynchronous Probing 7.5 Probe UDP and Obfuscated OpenVPN Servers 7.5 Probe UDP and Obfuscated OpenVPN Servers 8 Real-world Deployment Setup 8 Real-world Deployment Setup 9 Evaluation & Findings and 9.1 Results for control VPN flows 9 Evaluation & Findings and 9.1 Results for control VPN flows 9.2 Results for all flows 9.2 Results for all flows 10 Discussion and Mitigations 10 Discussion and Mitigations 11 Conclusion 11 Conclusion 12 Acknowledgement and References 12 Acknowledgement and References Appendix Appendix 6.2 ACK-based Fingerprinting OpenVPN engages in a TLS-style handshake with its peer over the control channel. Since TLS is designed to operate over a reliable layer, OpenVPN implements an explicit acknowledgement and re-transmission mechanism for its control channel messages [30]. Specifically, incoming P_- Control packets are acknowledged by P_ACK packets, which do not carry any TLS payloads and are uniform in size (Note these ACK packets are carried over by TCP as payload and are not the same as TCP ACK flags). Moreover, these ACK packets are seen mostly only in the early stage of a flow, during the handshake phase, and are not used in the actual data transfer channel, which can run over an unreliable layer. To our knowledge, we are the first to devise fingerprinting attempts based on the distinct protocol-layer ACKs against OpenVPN. Previously, the unique timing pattern in meek’s TCP-level ACK traffic has rendered the obfuscation tool vulnerable to detection [67]. For OpenVPN, the presence of explicit ACK packets, uniform in size and only seen in some parts of a session, provides another fingerprintable feature. Specifically, we first identify a likely ACK packet of a session by locating an initial packet exchange sequence of C- >S (Client-Reset), S->C (Server-Reset), C->S (ACK), C->S (Control), as illustrated in Figure 1. For vanilla OpenVPN and XOR-based obfuscation, the first ACK packet usually appears as the third (data) packet transmitted in a session. For tunnels or obfuscators that have their own handshake or key exchange process (e.g., Stunnel, SSH tunnel, or Obfsproxy), this counting is offset by the number of tunnel handshake packets. Next, we group packets into 10-packet bins, and we derive the ACK fingerprint for each flow by counting the number of packets in each bin that have the same size as the identified ACK packet. For OpenVPN flows, we expect to observe a high number of ACK packets in early bins and an absence of them in later bins. (Later in the session, Control and ACK packets can be exchanged again to transfer random key materials, but it is not expected to be observed within our observation window N.) This approach proves effective to fingerprint vanilla OpenVPN as well as obfuscated services running over encrypted tunnels that lack random padding. We quantify exact fingerprinting thresholds in Section 7.1. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Unveiling OpenVPN: Fingerprinting with ACK Patterns and Timing Analysis

Authors:
(1) Diwen Xue, University of Michigan;
(2) Reethika Ramesh, University of Michigan;
(3) Arham Jain, University of Michigan;
(4) Arham Jain, Merit Network, Inc.;
(5) J. Alex Halderman, University of Michigan;
(6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad;
(7) Roya Ensaf, University of Michigan. Table of Links Abstract and 1 Introduction 2 Background & Related Work 3 Challenges in Real-world VPN Detection 4 Adversary Model and Deployment 5 Ethics, Privacy, and Responsible Disclosure 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6.2 ACK-based Fingerprinting 6.3 Active Server Fingerprinting 6.4 Constructing Filters and Probers 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7.2 Choice of Observation Window N 7.3 Effects of Packet Loss 7.4 Server Churn for Asynchronous Probing 7.5 Probe UDP and Obfuscated OpenVPN Servers 8 Real-world Deployment Setup 9 Evaluation & Findings and 9.1 Results for control VPN flows 9.2 Results for all flows 10 Discussion and Mitigations 11 Conclusion 12 Acknowledgement and References Appendix 6 Identifying Fingerprintable Features In this section, we identify three features that fingerprint OpenVPN, exploiting byte pattern, packet length, and server behaviors, respectively. 6.1 Opcode-based Fingerprinting As shown in Figure 3, each OpenVPN packet has a header of 24 bits in TCP mode or 8 bits in UDP mode, which is not part of the encrypted payload. Each OpenVPN header starts with an opcode that specifies the message type of the current packet and a key ID that refers to a (new) TLS session. The opcode field can take over 10 defined values, corresponding to message types transmitted during different communication stages. A typical OpenVPN session starts with the client sending a Client Reset packet. The server then responds with a Server Reset packet, and a TLS handshake follows. OpenVPN packets that carry TLS ciphertexts have P_Control as their message type. Since OpenVPN can run over UDP but has to provide a reliable channel for TLS, each P_Control packet is explicitly acknowledged by P_ACK packets. Finally, actual payloads are transmitted as P_Data packets. Figure 1 illustrates this packet exchange with opcode annotations. A packet field taking a fixed number of values can be easy to fingerprint and has been exploited before against other protocols [1]. We fingerprint OpenVPN’s handshake sequence by analyzing each opcode byte for the first N packets of a flow (the threshold N is explored in Section 7.2). Algorithm 1 shows the process of opcode fingerprinting, with Opcode referring to the sequence of N opcode values found in the first N packets of a given flow. Briefly, the filter flags a flow if the number of different opcodes observed accords with the protocol and the Client and Server Resets are not seen once the handshake is completed. Previous work and existing open-source DPIs [23, 29, 37, 75] considered statically matching opcode values and packet sizes based on the protocol specification. In contrast, we propose to dynamically capture the variation in opcode values that reflects the establishment of OpenVPN sessions. Notably, our heuristics do not require exact matching of opcode values or packet length (e.g., do not require the third byte of the first packet to be 0x38), thereby ensuring it works effectively against XOR-obfuscated flows. The XOR obfuscation masks packet payloads to ensure that the opcode bytes are altered. Notably, according to the specification [36], when it reverses the packet as one of the obfuscation steps, it excludes the first character of the buffer (where the opcode byte is located) from reversal, as shown in Figure 4. As such, the opcode byte is always XOR-ed with the same byte of the XOR key, and the same opcodes would be mapped to the same value after obfuscation. This behavior is preserved when Tunnelblick (a number of unique opcodes seen so far, our heuristics are more flexible and target various XOR-based obfuscations of OpenVPN. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Diwen Xue, University of Michigan; (2) Reethika Ramesh, University of Michigan; (3) Arham Jain, University of Michigan; (4) Arham Jain, Merit Network, Inc.; (5) J. Alex Halderman, University of Michigan; (6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad; (7) Roya Ensaf, University of Michigan. Authors: Authors: (1) Diwen Xue, University of Michigan; (2) Reethika Ramesh, University of Michigan; (3) Arham Jain, University of Michigan; (4) Arham Jain, Merit Network, Inc.; (5) J. Alex Halderman, University of Michigan; (6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad; (7) Roya Ensaf, University of Michigan. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Background & Related Work 2 Background & Related Work 3 Challenges in Real-world VPN Detection 3 Challenges in Real-world VPN Detection 4 Adversary Model and Deployment 4 Adversary Model and Deployment 5 Ethics, Privacy, and Responsible Disclosure 5 Ethics, Privacy, and Responsible Disclosure 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6.2 ACK-based Fingerprinting 6.2 ACK-based Fingerprinting 6.3 Active Server Fingerprinting 6.3 Active Server Fingerprinting 6.4 Constructing Filters and Probers 6.4 Constructing Filters and Probers 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7.2 Choice of Observation Window N 7.2 Choice of Observation Window N 7.3 Effects of Packet Loss 7.3 Effects of Packet Loss 7.4 Server Churn for Asynchronous Probing 7.4 Server Churn for Asynchronous Probing 7.5 Probe UDP and Obfuscated OpenVPN Servers 7.5 Probe UDP and Obfuscated OpenVPN Servers 8 Real-world Deployment Setup 8 Real-world Deployment Setup 9 Evaluation & Findings and 9.1 Results for control VPN flows 9 Evaluation & Findings and 9.1 Results for control VPN flows 9.2 Results for all flows 9.2 Results for all flows 10 Discussion and Mitigations 10 Discussion and Mitigations 11 Conclusion 11 Conclusion 12 Acknowledgement and References 12 Acknowledgement and References Appendix Appendix 6 Identifying Fingerprintable Features In this section, we identify three features that fingerprint OpenVPN, exploiting byte pattern, packet length, and server behaviors, respectively. 6.1 Opcode-based Fingerprinting As shown in Figure 3, each OpenVPN packet has a header of 24 bits in TCP mode or 8 bits in UDP mode, which is not part of the encrypted payload. Each OpenVPN header starts with an opcode that specifies the message type of the current packet and a key ID that refers to a (new) TLS session. The opcode field can take over 10 defined values, corresponding to message types transmitted during different communication stages. A typical OpenVPN session starts with the client sending a Client Reset packet. The server then responds with a Server Reset packet, and a TLS handshake follows. OpenVPN packets that carry TLS ciphertexts have P_Control as their message type. Since OpenVPN can run over UDP but has to provide a reliable channel for TLS, each P_Control packet is explicitly acknowledged by P_ACK packets. Finally, actual payloads are transmitted as P_Data packets. Figure 1 illustrates this packet exchange with opcode annotations. A packet field taking a fixed number of values can be easy to fingerprint and has been exploited before against other protocols [1]. We fingerprint OpenVPN’s handshake sequence by analyzing each opcode byte for the first N packets of a flow (the threshold N is explored in Section 7.2). Algorithm 1 shows the process of opcode fingerprinting, with Opcode referring to the sequence of N opcode values found in the first N packets of a given flow. Briefly, the filter flags a flow if the number of different opcodes observed accords with the protocol and the Client and Server Resets are not seen once the handshake is completed. Previous work and existing open-source DPIs [23, 29, 37, 75] considered statically matching opcode values and packet sizes based on the protocol specification. In contrast, we propose to dynamically capture the variation in opcode values that reflects the establishment of OpenVPN sessions. Notably, our heuristics do not require exact matching of opcode values or packet length (e.g., do not require the third byte of the first packet to be 0x38), thereby ensuring it works effectively against XOR-obfuscated flows. The XOR obfuscation masks packet payloads to ensure that the opcode bytes are altered. Notably, according to the specification [36], when it reverses the packet as one of the obfuscation steps, it excludes the first character of the buffer (where the opcode byte is located) from reversal, as shown in Figure 4. As such, the opcode byte is always XOR-ed with the same byte of the XOR key, and the same opcodes would be mapped to the same value after obfuscation. This behavior is preserved when Tunnelblick (a number of unique opcodes seen so far, our heuristics are more flexible and target various XOR-based obfuscations of OpenVPN. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Identifying Fingerprintable Features and Opcode-based Fingerprinting

Authors:
(1) Diwen Xue, University of Michigan;
(2) Reethika Ramesh, University of Michigan;
(3) Arham Jain, University of Michigan;
(4) Arham Jain, Merit Network, Inc.;
(5) J. Alex Halderman, University of Michigan;
(6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad;
(7) Roya Ensaf, University of Michigan. Table of Links Abstract and 1 Introduction 2 Background & Related Work 3 Challenges in Real-world VPN Detection 4 Adversary Model and Deployment 5 Ethics, Privacy, and Responsible Disclosure 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6.2 ACK-based Fingerprinting 6.3 Active Server Fingerprinting 6.4 Constructing Filters and Probers 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7.2 Choice of Observation Window N 7.3 Effects of Packet Loss 7.4 Server Churn for Asynchronous Probing 7.5 Probe UDP and Obfuscated OpenVPN Servers 8 Real-world Deployment Setup 9 Evaluation & Findings and 9.1 Results for control VPN flows 9.2 Results for all flows 10 Discussion and Mitigations 11 Conclusion 12 Acknowledgement and References Appendix 2 Background & Related Work VPN tools create private networks across the public Internet through encrypted tunneling. Although many VPN protocols are being used, such as IPSec and WireGuard, OpenVPN remains the most supported and trusted protocol among commercial VPN providers [6]. Due to its versatility and opensource nature, OpenVPN has been used as the underlying protocol in numerous VPN products, which often advertise the protocol for its proven security [66]. In addition, OpenVPN’s popularity continues to rise with the trend of users choosing to self-host open-source VPN tools [65]. OpenVPN Protocol. OpenVPN was first released in 2002 with the aim of creating a tunneling protocol focusing on security, while also being free and fast over the standard TCP and UDP [34]. When the OpenVPN tunnel is active, raw IP packets being sent to or from the tunnel to the final destination are encapsulated inside OpenVPN packets. To achieve secure communication, OpenVPN leverages the OpenSSL library as its cryptographic layer. Two methods for authentication and key exchange are provided to establish trust with peers: either pre-shared static key(s) or TLS-based negotiations. The latter has been adopted by the majority of commercial VPN services. Two separate channels are used for key exchange and data transfer, both sharing a single multiplexed TCP/UDP stream. In the control channel, the client and server engage in a TLS-style exchange of key materials. As TLS is designed to operate over a reliable transport, OpenVPN provides its control channel with a sequential, reliable layer based on an explicit acknowledgement and re-transmission mechanism. The negotiated key from the control channel will be used to encrypt packets transferred in the data channel, which does not provide any reliability guarantee. Figure 1 presents a typical initialization sequence of OpenVPN packets leading to a fully encrypted data channel. Tor, Proxy, and VPN Detection. The ongoing arms-race between the GFW and Tor has been extensively studied and is most representative of the conflict between censorship & surveillance and circumvention tools [9, 11, 12, 55, 56, 71]. Censors started by blocking Tor’s website and public relays, which Tor responded to by deploying website mirrors and private, unpublished bridges. Next, censors moved to blocking with DPI by fingerprinting Tor’s TLS handshake, e.g. cipher suites. Tor used Pluggable Transports (PT) obfuscators, such as Obfsproxy and meek [39], to mask the handshake. In response, censors deployed active probing to complement DPI-based fingerprinting to detect Tor and certain obfuscators. There is limited previous work focusing on VPN traffic detection. Hoogstraaten [19] explored server-side VPN detection methods, ranging from using existing information databases (e.g. WHOIS, rDNS) to fingerprinting TCP options (e.g. advertised MSS). Webb et al. [70] proposed detecting proxies and VPNs based on traffic timing and latency. Their approach relied on the hypothesis that when a service is accessed through a proxy, the RTT measurement will be different from the RTT of a direct connection. Another class of previous work uses computational and machine learning models to passively detect VPN traffic [3,14,15,17,24,26,68], leveraging flow-level statistics such as connection duration and packet interval. Most of this work uses the same synthetic ISCXVPN2016 dataset [17]—which contains a balanced mixture of VPN and non-VPN traffic—to train and test a variety of machine learning and neural network classifiers in an offline, lab-setting. In contrast, our work primarily focuses on whether ISP-level adversaries can identify OpenVPN flows in near real time, and whether they can do so at scale, under practical constraints, and with minimal collateral damage. For this reason, we omit a full analysis of ML-based work, and only compare them with our approach in terms of false positives (falsely blocking legitimate traffic). Obfuscated (Open)VPN. Various traffic obfuscation techniques have been examined in previous work. Wang et al. examined the detectability of Obfsproxy, FTE, and meek [67]. Using attacks based on protocol semantics, packet entropy, and timing-related features, they concluded that a determined censor could detect all three obfuscators reliably. Houmansadr et al. demonstrated that popular mimicry-based obfuscation tools failed to achieve unobservability because seamlessly simulating another protocol is extremely challenging [20]. Previous studies have suggested censors can use active probing to detect proxies that obfuscate traffic [1,11,71]. In response, “probe-resistant” proxies were developed, which remain silent when being probed by an unauthenticated adversary. However, researchers have demonstrated that carefully designed probes could still identify these proxies [13]. There is a marked demand for an emerging class of services called “stealth” or “obfuscated” VPN, especially from users in countries with heavy censorship or laws against personal VPN usage [60, 63]. Most obfuscated VPN services use OpenVPN as the underlying protocol for security and routing, with an obfuscation layer overlaid to avoid detection [2, 66] [1]. OpenVPN’s core developers prefer that obfuscation remains a separate project operating alongside the vanilla/core protocol, as they “do not want to play the cat-and-mouse game [as Tor]” [35]. The absence of a standardized obfuscation solution has led to a plethora of obfuscators implemented by different VPN providers, who often claim that their obfuscated services can remain undetected by ISPs and censors alike. For example, TorGuard introduces their obfuscated VPN service as “Engineered from the ground up to be impossible to detect” [54]. BolehVPN claims that their VPN obfuscation “...keeps you out of trouble, even in China” [5]. Common obfuscation strategies adopted by commercial VPNs include employing XOR-based scramblers, wrapping OpenVPN inside encrypted tunnels, or using proprietary protocols. OpenVPN XOR Patch: Originally developed by Clayface as a patch for vanilla OpenVPN, the XOR patch scrambles a packet by either xor-ing bytes with a pre-shared key, reversing the order of the bytes, xor-ing each byte with its position, or a combination of these steps [36]. Notably, OpenVPN developers discourage its use due to the lack of code audit [57]. OpenVPN over Encrypted Tunnels: Some VPN services wrap OpenVPN traffic inside encrypted tunnels to prevent DPI fingerprinting. Some of the adopted obfuscation tunnels are Obfsproxy (obfs{2/3/4}), Stunnel, Websocket Tunnel, and encrypted proxies (shadowsocks, V2Ray). Proprietary Protocols: A few VPN providers have developed proprietary obfuscated protocols, some of which are built on top of OpenVPN with a proprietary obfuscation layer added, such as VyprVPN or Astrill [2, 66]. To the best of our knowledge, we are the first to explore the fingerprintability of commercial and/or obfuscated OpenVPN services on real traffic. Our unique study highlights the practicality of such fingerprinting, which has profound real-world security implications on end-users expecting certain privacy and anonymity guarantees from using these services. This paper is available on arxiv under CC BY 4.0 DEED license. [1] There are discussions on obfuscating WireGuard [72, 73], but to the best of our knowledge, they have yet to be deployed by any commercial VPNs Authors: (1) Diwen Xue, University of Michigan; (2) Reethika Ramesh, University of Michigan; (3) Arham Jain, University of Michigan; (4) Arham Jain, Merit Network, Inc.; (5) J. Alex Halderman, University of Michigan; (6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad; (7) Roya Ensaf, University of Michigan. Authors: Authors: (1) Diwen Xue, University of Michigan; (2) Reethika Ramesh, University of Michigan; (3) Arham Jain, University of Michigan; (4) Arham Jain, Merit Network, Inc.; (5) J. Alex Halderman, University of Michigan; (6) Jedidiah R. Crandall, Arizona State University/Breakpointing Bad; (7) Roya Ensaf, University of Michigan. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Background & Related Work 2 Background & Related Work 3 Challenges in Real-world VPN Detection 3 Challenges in Real-world VPN Detection 4 Adversary Model and Deployment 4 Adversary Model and Deployment 5 Ethics, Privacy, and Responsible Disclosure 5 Ethics, Privacy, and Responsible Disclosure 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6 Identifying Fingerprintable Features and 6.1 Opcode-based Fingerprinting 6.2 ACK-based Fingerprinting 6.2 ACK-based Fingerprinting 6.3 Active Server Fingerprinting 6.3 Active Server Fingerprinting 6.4 Constructing Filters and Probers 6.4 Constructing Filters and Probers 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7 Fine-tuning for Deployment and 7.1 ACK Fingerprint Thresholds 7.2 Choice of Observation Window N 7.2 Choice of Observation Window N 7.3 Effects of Packet Loss 7.3 Effects of Packet Loss 7.4 Server Churn for Asynchronous Probing 7.4 Server Churn for Asynchronous Probing 7.5 Probe UDP and Obfuscated OpenVPN Servers 7.5 Probe UDP and Obfuscated OpenVPN Servers 8 Real-world Deployment Setup 8 Real-world Deployment Setup 9 Evaluation & Findings and 9.1 Results for control VPN flows 9 Evaluation & Findings and 9.1 Results for control VPN flows 9.2 Results for all flows 9.2 Results for all flows 10 Discussion and Mitigations 10 Discussion and Mitigations 11 Conclusion 11 Conclusion 12 Acknowledgement and References 12 Acknowledgement and References Appendix Appendix 2 Background & Related Work VPN tools create private networks across the public Internet through encrypted tunneling. Although many VPN protocols are being used, such as IPSec and WireGuard, OpenVPN remains the most supported and trusted protocol among commercial VPN providers [6]. Due to its versatility and opensource nature, OpenVPN has been used as the underlying protocol in numerous VPN products, which often advertise the protocol for its proven security [66]. In addition, OpenVPN’s popularity continues to rise with the trend of users choosing to self-host open-source VPN tools [65]. OpenVPN Protocol. OpenVPN was first released in 2002 with the aim of creating a tunneling protocol focusing on security, while also being free and fast over the standard TCP and UDP [34]. When the OpenVPN tunnel is active, raw IP packets being sent to or from the tunnel to the final destination are encapsulated inside OpenVPN packets. To achieve secure communication, OpenVPN leverages the OpenSSL library as its cryptographic layer. Two methods for authentication and key exchange are provided to establish trust with peers: either pre-shared static key(s) or TLS-based negotiations. The latter has been adopted by the majority of commercial VPN services. Two separate channels are used for key exchange and data transfer, both sharing a single multiplexed TCP/UDP stream. In the control channel, the client and server engage in a TLS-style exchange of key materials. As TLS is designed to operate over a reliable transport, OpenVPN provides its control channel with a sequential, reliable layer based on an explicit acknowledgement and re-transmission mechanism. The negotiated key from the control channel will be used to encrypt packets transferred in the data channel, which does not provide any reliability guarantee. Figure 1 presents a typical initialization sequence of OpenVPN packets leading to a fully encrypted data channel. OpenVPN Protocol. Tor, Proxy, and VPN Detection. The ongoing arms-race between the GFW and Tor has been extensively studied and is most representative of the conflict between censorship & surveillance and circumvention tools [9, 11, 12, 55, 56, 71]. Censors started by blocking Tor’s website and public relays, which Tor responded to by deploying website mirrors and private, unpublished bridges. Next, censors moved to blocking with DPI by fingerprinting Tor’s TLS handshake, e.g. cipher suites. Tor used Pluggable Transports (PT) obfuscators, such as Obfsproxy and meek [39], to mask the handshake. In response, censors deployed active probing to complement DPI-based fingerprinting to detect Tor and certain obfuscators. Tor, Proxy, and VPN Detection. There is limited previous work focusing on VPN traffic detection. Hoogstraaten [19] explored server-side VPN detection methods, ranging from using existing information databases (e.g. WHOIS, rDNS) to fingerprinting TCP options (e.g. advertised MSS). Webb et al. [70] proposed detecting proxies and VPNs based on traffic timing and latency. Their approach relied on the hypothesis that when a service is accessed through a proxy, the RTT measurement will be different from the RTT of a direct connection. Another class of previous work uses computational and machine learning models to passively detect VPN traffic [3,14,15,17,24,26,68], leveraging flow-level statistics such as connection duration and packet interval. Most of this work uses the same synthetic ISCXVPN2016 dataset [17]—which contains a balanced mixture of VPN and non-VPN traffic—to train and test a variety of machine learning and neural network classifiers in an offline, lab-setting. In contrast, our work primarily focuses on whether ISP-level adversaries can identify OpenVPN flows in near real time, and whether they can do so at scale, under practical constraints, and with minimal collateral damage. For this reason, we omit a full analysis of ML-based work, and only compare them with our approach in terms of false positives (falsely blocking legitimate traffic). Obfuscated (Open)VPN. Various traffic obfuscation techniques have been examined in previous work. Wang et al. examined the detectability of Obfsproxy, FTE, and meek [67]. Using attacks based on protocol semantics, packet entropy, and timing-related features, they concluded that a determined censor could detect all three obfuscators reliably. Houmansadr et al. demonstrated that popular mimicry-based obfuscation tools failed to achieve unobservability because seamlessly simulating another protocol is extremely challenging [20]. Previous studies have suggested censors can use active probing to detect proxies that obfuscate traffic [1,11,71]. In response, “probe-resistant” proxies were developed, which remain silent when being probed by an unauthenticated adversary. However, researchers have demonstrated that carefully designed probes could still identify these proxies [13]. Obfuscated (Open)VPN. There is a marked demand for an emerging class of services called “stealth” or “obfuscated” VPN, especially from users in countries with heavy censorship or laws against personal VPN usage [60, 63]. Most obfuscated VPN services use OpenVPN as the underlying protocol for security and routing, with an obfuscation layer overlaid to avoid detection [2, 66] [1]. OpenVPN’s core developers prefer that obfuscation remains a separate project operating alongside the vanilla/core protocol, as they “do not want to play the cat-and-mouse game [as Tor]” [35]. The absence of a standardized obfuscation solution has led to a plethora of obfuscators implemented by different VPN providers, who often claim that their obfuscated services can remain undetected by ISPs and censors alike. For example, TorGuard introduces their obfuscated VPN service as “Engineered from the ground up to be impossible to detect” [54]. BolehVPN claims that their VPN obfuscation “...keeps you out of trouble, even in China” [5]. Common obfuscation strategies adopted by commercial VPNs include employing XOR-based scramblers, wrapping OpenVPN inside encrypted tunnels, or using proprietary protocols. OpenVPN XOR Patch: Originally developed by Clayface as a patch for vanilla OpenVPN, the XOR patch scrambles a packet by either xor-ing bytes with a pre-shared key, reversing the order of the bytes, xor-ing each byte with its position, or a combination of these steps [36]. Notably, OpenVPN developers discourage its use due to the lack of code audit [57]. OpenVPN XOR Patch: OpenVPN over Encrypted Tunnels: Some VPN services wrap OpenVPN traffic inside encrypted tunnels to prevent DPI fingerprinting. Some of the adopted obfuscation tunnels are Obfsproxy (obfs{2/3/4}), Stunnel, Websocket Tunnel, and encrypted proxies (shadowsocks, V2Ray). OpenVPN over Encrypted Tunnels: Proprietary Protocols: A few VPN providers have developed proprietary obfuscated protocols, some of which are built on top of OpenVPN with a proprietary obfuscation layer added, such as VyprVPN or Astrill [2, 66]. Proprietary Protocols: To the best of our knowledge, we are the first to explore the fingerprintability of commercial and/or obfuscated OpenVPN services on real traffic. Our unique study highlights the practicality of such fingerprinting, which has profound real-world security implications on end-users expecting certain privacy and anonymity guarantees from using these services. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv [1] There are discussions on obfuscating WireGuard [72, 73], but to the best of our knowledge, they have yet to be deployed by any commercial VPNs

Study Reveals OpenVPN Fingerprintability, Threatening VPN Privacy Guarantees

New Study Exposes OpenVPN Fingerprintability, Raising Privacy Concerns

Новае даследаванне раскрывае магчымасць адбіткаў пальцаў OpenVPN, што выклікае занепакоенасць прыватнасцю

Ново проучване разкрива възможността за пръстови отпечатъци на OpenVPN, което поражда опасения за поверителността

Νέα μελέτη αποκαλύπτει τη δυνατότητα δακτυλικών αποτυπωμάτων του OpenVPN, εγείροντας ανησυχίες για το απόρρητο

Un nuevo estudio revela la posibilidad de que OpenVPN sea objeto de huellas dactilares, lo que genera inquietudes sobre la privacidad

नए अध्ययन से ओपनवीपीएन की फिंगरप्रिंटेबिलिटी उजागर हुई, जिससे गोपनीयता संबंधी चिंताएं बढ़ीं

新たな研究で OpenVPN のフィンガープリント可能性が明らかに、プライバシーへの懸念が高まる

ការសិក្សាថ្មីបង្ហាញពីសមត្ថភាពស្កេនក្រយៅដៃ OpenVPN បង្កើនការព្រួយបារម្ភអំពីឯកជនភាព

Шинэ судалгаа нь OpenVPN-ийн хурууны хээг ил болгож, нууцлалын асуудалд анхаарал хандуулж байна

نوې مطالعه د OpenVPN د ګوتو چاپ وړتیا څرګندوي، د محرمیت اندیښنې راپورته کوي

Inyigisho nshya Yerekana Gufungura Urutoki rwa VPN, Kuzamura Ibanga

Kudzidza Kutsva Kunofumura OpenVPN Fingerprintability, Kusimudza Zvekuvanzika Zvinonetsa

Ny studie avslöjar OpenVPN-fingeravtryck, vilket väcker integritetsproblem

Your web scraper got blocked again? Ugh, what now? You nailed those HTTP headers and made it look just like a browser, but the site still figured out your requests were automated. How’s that even possible? Simple: it’s your TLS fingerprint! 😲 Dive into the sneaky world of TLS fingerprinting, uncover why it’s the silent killer behind most blocks, and learn you how to get around it. Anti-Bot Blocked You Again? Time to Learn Why! Let’s assume you’re dealing with a typical scraping scenario. You're making an automated request using an HTTP client—like Requests in Python or Axios in JavaScript—to fetch the HTML of a web page to scrape some data from it. As you probably already know, most websites have bot protection technologies in place. Curious about the best anti-scraping tech? Check our guide on the best anti-scraping solutions! 🔐 These tools monitor incoming requests, filtering out the suspicious ones. If your request looks like it’s coming from a regular human, you're good to go. Otherwise? It’s going to get stonewalled! 🧱 Browser Requests vs Bot Requests Now, what does a request from a regular user look like? Easy! Just fire your browser's DevTools, head to the Network tab, and see for yourself: If you copy that request to cURL by selecting the option from the right-click menu, you'll get something like this: curl 'https://kick.com/emotes/ninja' \
  -H 'accept: application/json' \
  -H 'accept-language: en-US,en;q=0.9' \
  -H 'cache-control: max-age=0' \
  -H 'cluster: v1' \
  -H 'priority: u=1, i' \
  -H 'referer: https://kick.com/ninja' \
  -H 'sec-ch-ua: "Google Chrome";v="129", "Not=A?Brand";v="8", "Chromium";v="129"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "Windows"' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: same-origin' \
  -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36' If this syntax looks like Chinese to you, no worries—check out our introduction to cURL. 📖 Basically, a “human” request is just a regular HTTP request with some extra headers (the -H flags). Anti-bot systems inspect those headers to figure out if a request is coming from a bot or a legit user in a browser. One of their biggest red flags? The User-Agent header! Explore our post on the best user agents for web scraping. That header is automatically set by HTTP clients but never quite matches the ones used by real browsers. Mismatch in those headers? It’s a dead giveaway for bots! 💀 For more information, dive into our guide on HTTP headers for web scraping. Setting HTTP Headers Isn’t Always the Solution Now, you might be thinking: “Easy fix, I’ll just perform automated requests with those headers!” But hold on a sec… 🚨 Go ahead and run that cURL request you copied from DevTools: Surprise! The server hit you back with a “403 Access Denied” page from Cloudflare. Yep, even with the browser-like headers, you can still get blocked! Cracking Cloudflare isn’t that easy, after all. 😅 But wait, how?! Isn’t that the exact same request a browser would make? 🤔 Well, not quite… The Key Lies in the OSI Model On the application level of the OSI Model, the browser and cURL requests are the same. Yet, there are all underlying layers you might be overlooking. 🫠 Some of these layers are often the culprits behind those pesky blocks, and information transferred there is exactly what advanced anti-scraping technologies focus on. Sly sneaky beasts! 👹 For instance, they look at your IP address, which is pulled from the Network layer. Want to dodge those IP bans? Follow our tutorial on how to avoid an IP ban with proxies! Unfortunately, that’s not all! 😩 Anti-bot systems also pay close attention to the TLS fingerprint from the secure communication channel established between your script and the target web server at the Transport Layer. That’s where things differ between a browser and an automated HTTP request! Cool, right? But now you must be wondering what that entails… 🔍 What’s a TLS Fingerprint? A TLS fingerprint is a unique identifier that anti-bot solutions create when your browser or HTTP client sets up a secure connection to a website. It’s like a digital signature your machine leaves behind during the TLS handshake—the initial “conversation” between a client and the web server to decide how they’ll encrypt and secure data at the Transport layer. 🤝 When you make an HTTP request to a site, the underlying TLS library in your browser or HTTP client kicks off the handshake procedure. The two parties, the client and the server, start asking each other things like, “What encryption protocols do you support?” and “Which ciphers should we use?” ❓ Based on your answers, the server can tell if you’re a regular user in a browser or an automated script using an HTTP client. In other words, if your answers don’t match those of typical browsers, you might get blocked. Imagine this handshake like two people meeting: Human version: Server: "What language do you speak?"


Browser: "English, French, Chinese, and Spanish"


Server: "Great, let’s chat" Bot version: Server: "What language do you speak?"


Bot: “Meow! 🐈”


Server: “Sorry, but you don’t seem like a human being. Blocked!" TLS fingerprinting operates below the Application layer of the OSI model. That means you can’t just tweak your TLS fingerprint with a few lines of code. 🚫 💻 🚫 To spoof TLS fingerprints, you need to swap your HTTP client’s TLS configurations with those of a real browser. The catch? Not all HTTP clients let you do this! That’s where tools like cURL Impersonate come into play. This special build of cURL is designed to mimic a browser’s TLS settings, helping you simulate a browser from the command line! Why a Headless Browser May Not Be a Solution Either Now, you might be thinking: “Well, if HTTP clients give off ‘bot-like’ TLS fingerprints, why not just use a browser for scraping?” The idea is to use a browser automation tool to run specific tasks on a webpage with a headless browser. Whether the browser runs in headed or headless mode, it still uses the same underlying TLS libraries. That’s good news because it means headless browsers generate a "human-like" TLS fingerprint! 🎉 That’s the solution, right? Not really… 🫤 Here’s the kicker: headless browsers come with other configurations that scream, “I’m a bot!” 🤖 Sure, you could try hiding that with a stealth plugin in Puppeteer Extra, but advanced anti-bot systems can still sniff out headless browsers through JavaScript challenges and browser fingerprinting. So, yeah, headless browsers aren’t your foolproof escape either to anti-bots. 😬 How to Really Bypass TLS Fingerprinting TLS fingerprint checking is just one of many advanced bot protection tactics that sophisticated anti-scraping solutions implement. 🛡️ To truly leave behind the headaches of TLS fingerprinting and other annoying blocks, you need a next-level scraping solution that provides: Reliable TLS fingerprints


Unlimited scalability


CAPTCHA-solving superpowers


Built-in IP rotation via a 72-million IP proxy network


Automatic retries


JavaScript rendering capabilities Those are some of the many features offered by Bright Data's Scraping Browser API—an all-in-one cloud browser solution to scrape the Web efficiently and effectively. This product integrates seamlessly with your favorite browser automation tools, including Playwright, Selenium, and Puppeteer. ✨ Just set up the automation logic, run your script, and let the Scraping Browser API handle the dirty work. Forget about blocks and get back to what matters—scraping at full speed! ⚡️ https://www.youtube.com/watch?v=21Xyi1HMTng&embedable=true Don’t need to interact with the page? Try Bright Data’s Web Unlocker! Final Thoughts Now you finally know why working at the application level isn’t enough to avoid all blocks. The TLS library your HTTP client uses plays a big part, too. TLS fingerprinting? No longer a mystery—you’ve cracked it and know how to tackle it. Looking for a way to scrape without hitting blocks? Look no further than Bright Data's suite of tools! Join the mission to make the Internet accessible to all—even via automated HTTP requests. 🌐 Until next time, keep surfing the Web with freedom! Your web scraper got blocked again? Ugh, what now? You nailed those HTTP headers and made it look just like a browser, but the site still figured out your requests were automated. How’s that even possible? Simple: it’s your TLS fingerprint! 😲 Dive into the sneaky world of TLS fingerprinting, uncover why it’s the silent killer behind most blocks, and learn you how to get around it. Anti-Bot Blocked You Again? Time to Learn Why! Let’s assume you’re dealing with a typical scraping scenario. You're making an automated request using an HTTP client—like Requests in Python or Axios in JavaScript—to fetch the HTML of a web page to scrape some data from it. Requests Axios As you probably already know, most websites have bot protection technologies in place. Curious about the best anti-scraping tech? Check our guide on the best anti-scraping solutions! 🔐 bot protection technologies These tools monitor incoming requests, filtering out the suspicious ones. If your request looks like it’s coming from a regular human, you're good to go. Otherwise? It’s going to get stonewalled! 🧱 Browser Requests vs Bot Requests Now, what does a request from a regular user look like? Easy! Just fire your browser's DevTools, head to the Network tab, and see for yourself: If you copy that request to cURL by selecting the option from the right-click menu, you'll get something like this: curl 'https://kick.com/emotes/ninja' \
  -H 'accept: application/json' \
  -H 'accept-language: en-US,en;q=0.9' \
  -H 'cache-control: max-age=0' \
  -H 'cluster: v1' \
  -H 'priority: u=1, i' \
  -H 'referer: https://kick.com/ninja' \
  -H 'sec-ch-ua: "Google Chrome";v="129", "Not=A?Brand";v="8", "Chromium";v="129"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "Windows"' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: same-origin' \
  -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36' curl 'https://kick.com/emotes/ninja' \
  -H 'accept: application/json' \
  -H 'accept-language: en-US,en;q=0.9' \
  -H 'cache-control: max-age=0' \
  -H 'cluster: v1' \
  -H 'priority: u=1, i' \
  -H 'referer: https://kick.com/ninja' \
  -H 'sec-ch-ua: "Google Chrome";v="129", "Not=A?Brand";v="8", "Chromium";v="129"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "Windows"' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: same-origin' \
  -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36' If this syntax looks like Chinese to you, no worries—check out our introduction to cURL . 📖 introduction to cURL Basically, a “human” request is just a regular HTTP request with some extra headers (the -H flags). Anti-bot systems inspect those headers to figure out if a request is coming from a bot or a legit user in a browser. -H One of their biggest red flags? The User-Agent header! Explore our post on the best user agents for web scraping . That header is automatically set by HTTP clients but never quite matches the ones used by real browsers. User-Agent best user agents for web scraping Mismatch in those headers? It’s a dead giveaway for bots! 💀 For more information, dive into our guide on HTTP headers for web scraping . HTTP headers for web scraping Setting HTTP Headers Isn’t Always the Solution Now, you might be thinking: “Easy fix, I’ll just perform automated requests with those headers!” But hold on a sec… 🚨 Go ahead and run that cURL request you copied from DevTools: Surprise! The server hit you back with a “403 Access Denied” page from Cloudflare. Yep, even with the browser-like headers, you can still get blocked! Cracking Cloudflare isn’t that easy, after all. 😅 Cracking Cloudflare But wait, how?! Isn’t that the exact same request a browser would make? 🤔 Well, not quite… The Key Lies in the OSI Model On the application level of the OSI Model, the browser and cURL requests are the same. Yet, there are all underlying layers you might be overlooking. 🫠 Some of these layers are often the culprits behind those pesky blocks, and information transferred there is exactly what advanced anti-scraping technologies focus on. Sly sneaky beasts! 👹 For instance, they look at your IP address , which is pulled from the Network layer. Want to dodge those IP bans? Follow our tutorial on how to avoid an IP ban with proxies ! IP address how to avoid an IP ban with proxies Unfortunately, that’s not all! 😩 Anti-bot systems also pay close attention to the TLS fingerprint from the secure communication channel established between your script and the target web server at the Transport Layer. TLS That’s where things differ between a browser and an automated HTTP request! Cool, right? But now you must be wondering what that entails… 🔍 What’s a TLS Fingerprint? A TLS fingerprint is a unique identifier that anti-bot solutions create when your browser or HTTP client sets up a secure connection to a website. TLS fingerprint It’s like a digital signature your machine leaves behind during the TLS handshake —the initial “conversation” between a client and the web server to decide how they’ll encrypt and secure data at the Transport layer. 🤝 TLS handshake When you make an HTTP request to a site, the underlying TLS library in your browser or HTTP client kicks off the handshake procedure. The two parties, the client and the server, start asking each other things like, “What encryption protocols do you support?” and “Which ciphers should we use?” ❓ Based on your answers, the server can tell if you’re a regular user in a browser or an automated script using an HTTP client. In other words, if your answers don’t match those of typical browsers, you might get blocked. Imagine this handshake like two people meeting: Human version : Human version Server: "What language do you speak?" Browser: "English, French, Chinese, and Spanish" Server: "Great, let’s chat" Server: "What language do you speak?" Server: "What language do you speak?" Browser: "English, French, Chinese, and Spanish" Browser: "English, French, Chinese, and Spanish" Server: "Great, let’s chat" Server: "Great, let’s chat" Bot version : Bot version Server: "What language do you speak?" Bot: “Meow! 🐈” Server: “Sorry, but you don’t seem like a human being. Blocked!" Server: "What language do you speak?" Server: "What language do you speak?" Bot: “Meow! 🐈” Bot: “Meow! 🐈” Server: “Sorry, but you don’t seem like a human being. Blocked!" Server: “Sorry, but you don’t seem like a human being. Blocked!" TLS fingerprinting operates below the Application layer of the OSI model. That means you can’t just tweak your TLS fingerprint with a few lines of code. 🚫 💻 🚫 To spoof TLS fingerprints, you need to swap your HTTP client’s TLS configurations with those of a real browser. The catch? Not all HTTP clients let you do this! That’s where tools like cURL Impersonate come into play. This special build of cURL is designed to mimic a browser’s TLS settings, helping you simulate a browser from the command line! cURL Impersonate Why a Headless Browser May Not Be a Solution Either Now, you might be thinking: “Well, if HTTP clients give off ‘bot-like’ TLS fingerprints, why not just use a browser for scraping?” The idea is to use a browser automation tool to run specific tasks on a webpage with a headless browser. browser automation Whether the browser runs in headed or headless mode, it still uses the same underlying TLS libraries. That’s good news because it means headless browsers generate a "human-like" TLS fingerprint! 🎉 That’s the solution, right? Not really… 🫤 Here’s the kicker: headless browsers come with other configurations that scream, “I’m a bot!” 🤖 Sure, you could try hiding that with a stealth plugin in Puppeteer Extra , but advanced anti-bot systems can still sniff out headless browsers through JavaScript challenges and browser fingerprinting. stealth plugin in Puppeteer Extra So, yeah, headless browsers aren’t your foolproof escape either to anti-bots. 😬 How to Really Bypass TLS Fingerprinting TLS fingerprint checking is just one of many advanced bot protection tactics that sophisticated anti-scraping solutions implement. 🛡️ To truly leave behind the headaches of TLS fingerprinting and other annoying blocks, you need a next-level scraping solution that provides: Reliable TLS fingerprints Unlimited scalability CAPTCHA-solving superpowers Built-in IP rotation via a 72-million IP proxy network Automatic retries JavaScript rendering capabilities Reliable TLS fingerprints Reliable TLS fingerprints Unlimited scalability Unlimited scalability CAPTCHA-solving superpowers CAPTCHA-solving superpowers Built-in IP rotation via a 72-million IP proxy network Built-in IP rotation via a 72-million IP proxy network Automatic retries Automatic retries JavaScript rendering capabilities JavaScript rendering capabilities Those are some of the many features offered by Bright Data's Scraping Browser API —an all-in-one cloud browser solution to scrape the Web efficiently and effectively. Scraping Browser API This product integrates seamlessly with your favorite browser automation tools, including Playwright, Selenium, and Puppeteer. ✨ Just set up the automation logic, run your script, and let the Scraping Browser API handle the dirty work. Forget about blocks and get back to what matters—scraping at full speed! ⚡️ https://www.youtube.com/watch?v=21Xyi1HMTng&embedable=true https://www.youtube.com/watch?v=21Xyi1HMTng&embedable=true Don’t need to interact with the page? Try Bright Data’s Web Unlocker ! Web Unlocker Final Thoughts Now you finally know why working at the application level isn’t enough to avoid all blocks. The TLS library your HTTP client uses plays a big part, too. TLS fingerprinting? No longer a mystery—you’ve cracked it and know how to tackle it. Looking for a way to scrape without hitting blocks? Look no further than Bright Data 's suite of tools! Join the mission to make the Internet accessible to all—even via automated HTTP requests. 🌐 Bright Data Until next time, keep surfing the Web with freedom!

The Role of the TLS Fingerprint in Web Scraping

Ролята на пръстовия отпечатък на TLS в уеб скрапинга

El papel de la huella digital TLS en el web scraping

TLS hatz-markaren eginkizuna Web Scraping-en

TLS մատնահետքի դերը վեբ քերիչում

WebスクレイピングにおけるTLSフィンガープリントの役割

TLS pirksta nospieduma loma tīmekļa skrāpēšanā

Karolo ya Kgatišo ya Menwana ya TLS go go Scraping ya Wepe

Rolul amprentei TLS în Web Scraping

Úloha odtlačku prsta TLS pri zoškrabovaní webu

Jukumu la Alama ya Vidole ya TLS katika Uchakataji Wavuti

Роль TLS Fingerprint у веб-збиранні

Indima ye-TLS Fingerprint kwi-Web Scraping

I was utterly frustrated with scraping HTTP data from a firewall-protected website. Despite using residential proxies from multiple providers, my requests kept getting blocked without any clear reason. Sometimes, the script worked on my local machine, but it would fail when running on a cloud server. After extensive research, I stumbled upon the concept of TLS fingerprinting. Let me break it down for you: Understanding TLS Fingerprinting When we send an HTTPS request to a server, the process begins with a “Client Hello” TLS request. This request shares supported TLS versions, various cipher suites (encryption algorithms), the user agent, and several other parameters. These details and additional parameters create a unique TLS fingerprint for each request, making it easy to distinguish bots and scripts from legitimate browser clients. Check your TLS Fingerprint: https://tls.peet.ws/ Sources: https://www.zenrows.com/blog/what-is-tls-fingerprint The Search for a Solution Armed with this knowledge, I dug deeper into bypassing TLS fingerprinting. I couldn’t find a one-stop solution, so I’m sharing my findings. We’ll be using JavaScript to code the solution, but the actual request will be made using the curl command. This approach can be easily implemented in various programming languages. Initial Curl Command Here is the sample curl command I initially executed: curl --location 'url' \
--header 'accept: application/json, text/javascript, */*; q=0.01' \
--header 'accept-language: en-GB,en-US;q=0.9,en;q=0.8' \
--header 'cookie: authCookie' \
--header 'sec-ch-ua: "Not_A Brand";v="8", "Chromium";v="120", "Google Chrome";v="120"' \
--header 'sec-ch-ua-bitness: "64"' \
--header 'sec-ch-ua-full-version: "120.0.6099.234"' \
--header 'sec-ch-ua-full-version-list: "Not_A Brand";v="8.0.0.0", "Chromium";v="120.0.6099.234", "Google Chrome";v="120.0.6099.234"' \
--header 'sec-fetch-dest: empty' \
--header 'sec-fetch-mode: cors' \
--header 'sec-fetch-site: same-origin' \
--header 'user-agent: userAgent' \
--header 'x-requested-with: XMLHttpRequest' You might notice that I’m already using a proxy in the request. However, it wasn’t good enough, so I added a few things based on the User-Agent my script was creating: TLS Version
Cipher Suites Implementing the Solution Here is the JavaScript code snippet that provides the TLS version string for the request and, based on the version, selects and randomizes the Cipher Suites to prevent TLS fingerprinting:javascriptCopy code const tls12CipherSuites = [
  'ECDHE-ECDSA-AES256-GCM-SHA384',
  'ECDHE-RSA-AES256-GCM-SHA384',
  'ECDHE-ECDSA-AES128-GCM-SHA256',
  'ECDHE-RSA-AES128-GCM-SHA256',
  'ECDHE-ECDSA-AES256-SHA384',
  'ECDHE-RSA-AES256-SHA384',
  'ECDHE-ECDSA-AES128-SHA256',
  'ECDHE-RSA-AES128-SHA256',
  'DHE-RSA-AES256-GCM-SHA384',
  'DHE-RSA-AES128-GCM-SHA256',
  'DHE-RSA-AES256-SHA256',
  'DHE-RSA-AES128-SHA256',
];

const tls11CipherSuites = [
  'RSA-AES128-SHA',
  'RSA-AES256-SHA',
  'RSA-3DES-EDE-CBC-SHA',
  'ECDHE-RSA-AES128-SHA',
  'ECDHE-RSA-AES256-SHA',
  'ECDHE-ECDSA-AES128-SHA',
  'ECDHE-ECDSA-AES256-SHA',
  'DHE-RSA-AES128-SHA',
  'DHE-RSA-AES256-SHA',
  'DHE-RSA-3DES-EDE-CBC-SHA',
];

type SuiteMap = {
  [key: string]: string[];
};

const versionCipherMap: SuiteMap = {
  'v1.1': tls11CipherSuites,
  'v1.2': tls12CipherSuites,
};

function shuffledSuites(inputArray: string[]) {
  const firstThree = inputArray.slice(0, 3);
  const rest = inputArray.slice(3);
  for (let i = rest.length - 1; i > 0; i--) {
    const j = Math.floor(Math.random() * (i + 1));
    [rest[i], rest[j]] = [rest[j], rest[i]];
  }
  return firstThree.concat(rest).join(':');
}

function provideTLSAndSuites() {
  const tlsVersions = Object.keys(versionCipherMap);
  const selectedVersion = tlsVersions[Math.floor(Math.random() * tlsVersions.length)];
  const tlsVersionString = `--tls${selectedVersion}`;
  const shuffledSuitesString = shuffledSuites(versionCipherMap[selectedVersion]);
  return [tlsVersionString, shuffledSuitesString];
} This code provides the TLS version string for the request and selects and randomizes the cipher suites to prevent TLS fingerprinting. Final Curl Command We’ll fetch the TLS version and cipher suites for the request and embed them in the curl command: const [tlsVersionString, cipherSuitesString] = provideTLSAndSuites(); Here is the resulting curl command: curl --connect-timeout 30 --max-time 50 'url' \
 --proxy 'proxy_url' tlsVersionString --cipher cipherSuitesString \
 -H 'accept: application/json, text/javascript, */*; q=0.01' \
 -H 'accept-language: en-GB,en-US;q=0.9,en;q=0.8' \
 -H 'cookie: authCookie' \
 -H 'sec-ch-ua: "Not_A Brand";v="8", "Chromium";v="120", "Google Chrome";v="120"' \
 -H 'sec-ch-ua-bitness: "64"' \
 -H 'sec-ch-ua-full-version: "120.0.6099.234"' \
 -H 'sec-ch-ua-full-version-list: "Not_A Brand";v="8.0.0.0", "Chromium";v="120.0.6099.234", "Google Chrome";v="120.0.6099.234"' \
 -H 'sec-fetch-dest: empty' \
 -H 'sec-fetch-mode: cors' \
 -H 'sec-fetch-site: same-origin' \
 -H 'user-agent: userAgent' \
 -H 'x-requested-with: XMLHttpRequest' Conclusion Hopefully, this will allow your request to make it through the firewall. If this helped you or you enjoyed the content, don’t forget to clap and follow for more such content. Happy scraping! I was utterly frustrated with scraping HTTP data from a firewall-protected website. Despite using residential proxies from multiple providers, my requests kept getting blocked without any clear reason. Sometimes, the script worked on my local machine, but it would fail when running on a cloud server. After extensive research, I stumbled upon the concept of TLS fingerprinting. Let me break it down for you: Understanding TLS Fingerprinting When we send an HTTPS request to a server, the process begins with a “Client Hello” TLS request. This request shares supported TLS versions, various cipher suites (encryption algorithms), the user agent, and several other parameters. These details and additional parameters create a unique TLS fingerprint for each request, making it easy to distinguish bots and scripts from legitimate browser clients. Check your TLS Fingerprint: https://tls.peet.ws/ https://tls.peet.ws/ Sources: https://www.zenrows.com/blog/what-is-tls-fingerprint https://www.zenrows.com/blog/what-is-tls-fingerprint The Search for a Solution Armed with this knowledge, I dug deeper into bypassing TLS fingerprinting. I couldn’t find a one-stop solution, so I’m sharing my findings. We’ll be using JavaScript to code the solution, but the actual request will be made using the curl command. This approach can be easily implemented in various programming languages. curl Initial Curl Command Here is the sample curl command I initially executed: curl curl --location 'url' \
--header 'accept: application/json, text/javascript, */*; q=0.01' \
--header 'accept-language: en-GB,en-US;q=0.9,en;q=0.8' \
--header 'cookie: authCookie' \
--header 'sec-ch-ua: "Not_A Brand";v="8", "Chromium";v="120", "Google Chrome";v="120"' \
--header 'sec-ch-ua-bitness: "64"' \
--header 'sec-ch-ua-full-version: "120.0.6099.234"' \
--header 'sec-ch-ua-full-version-list: "Not_A Brand";v="8.0.0.0", "Chromium";v="120.0.6099.234", "Google Chrome";v="120.0.6099.234"' \
--header 'sec-fetch-dest: empty' \
--header 'sec-fetch-mode: cors' \
--header 'sec-fetch-site: same-origin' \
--header 'user-agent: userAgent' \
--header 'x-requested-with: XMLHttpRequest' curl --location 'url' \
--header 'accept: application/json, text/javascript, */*; q=0.01' \
--header 'accept-language: en-GB,en-US;q=0.9,en;q=0.8' \
--header 'cookie: authCookie' \
--header 'sec-ch-ua: "Not_A Brand";v="8", "Chromium";v="120", "Google Chrome";v="120"' \
--header 'sec-ch-ua-bitness: "64"' \
--header 'sec-ch-ua-full-version: "120.0.6099.234"' \
--header 'sec-ch-ua-full-version-list: "Not_A Brand";v="8.0.0.0", "Chromium";v="120.0.6099.234", "Google Chrome";v="120.0.6099.234"' \
--header 'sec-fetch-dest: empty' \
--header 'sec-fetch-mode: cors' \
--header 'sec-fetch-site: same-origin' \
--header 'user-agent: userAgent' \
--header 'x-requested-with: XMLHttpRequest' You might notice that I’m already using a proxy in the request. However, it wasn’t good enough, so I added a few things based on the User-Agent my script was creating: TLS Version Cipher Suites TLS Version Cipher Suites Implementing the Solution Here is the JavaScript code snippet that provides the TLS version string for the request and, based on the version, selects and randomizes the Cipher Suites to prevent TLS fingerprinting:javascriptCopy code const tls12CipherSuites = [
  'ECDHE-ECDSA-AES256-GCM-SHA384',
  'ECDHE-RSA-AES256-GCM-SHA384',
  'ECDHE-ECDSA-AES128-GCM-SHA256',
  'ECDHE-RSA-AES128-GCM-SHA256',
  'ECDHE-ECDSA-AES256-SHA384',
  'ECDHE-RSA-AES256-SHA384',
  'ECDHE-ECDSA-AES128-SHA256',
  'ECDHE-RSA-AES128-SHA256',
  'DHE-RSA-AES256-GCM-SHA384',
  'DHE-RSA-AES128-GCM-SHA256',
  'DHE-RSA-AES256-SHA256',
  'DHE-RSA-AES128-SHA256',
];

const tls11CipherSuites = [
  'RSA-AES128-SHA',
  'RSA-AES256-SHA',
  'RSA-3DES-EDE-CBC-SHA',
  'ECDHE-RSA-AES128-SHA',
  'ECDHE-RSA-AES256-SHA',
  'ECDHE-ECDSA-AES128-SHA',
  'ECDHE-ECDSA-AES256-SHA',
  'DHE-RSA-AES128-SHA',
  'DHE-RSA-AES256-SHA',
  'DHE-RSA-3DES-EDE-CBC-SHA',
];

type SuiteMap = {
  [key: string]: string[];
};

const versionCipherMap: SuiteMap = {
  'v1.1': tls11CipherSuites,
  'v1.2': tls12CipherSuites,
};

function shuffledSuites(inputArray: string[]) {
  const firstThree = inputArray.slice(0, 3);
  const rest = inputArray.slice(3);
  for (let i = rest.length - 1; i > 0; i--) {
    const j = Math.floor(Math.random() * (i + 1));
    [rest[i], rest[j]] = [rest[j], rest[i]];
  }
  return firstThree.concat(rest).join(':');
}

function provideTLSAndSuites() {
  const tlsVersions = Object.keys(versionCipherMap);
  const selectedVersion = tlsVersions[Math.floor(Math.random() * tlsVersions.length)];
  const tlsVersionString = `--tls${selectedVersion}`;
  const shuffledSuitesString = shuffledSuites(versionCipherMap[selectedVersion]);
  return [tlsVersionString, shuffledSuitesString];
} const tls12CipherSuites = [
  'ECDHE-ECDSA-AES256-GCM-SHA384',
  'ECDHE-RSA-AES256-GCM-SHA384',
  'ECDHE-ECDSA-AES128-GCM-SHA256',
  'ECDHE-RSA-AES128-GCM-SHA256',
  'ECDHE-ECDSA-AES256-SHA384',
  'ECDHE-RSA-AES256-SHA384',
  'ECDHE-ECDSA-AES128-SHA256',
  'ECDHE-RSA-AES128-SHA256',
  'DHE-RSA-AES256-GCM-SHA384',
  'DHE-RSA-AES128-GCM-SHA256',
  'DHE-RSA-AES256-SHA256',
  'DHE-RSA-AES128-SHA256',
];

const tls11CipherSuites = [
  'RSA-AES128-SHA',
  'RSA-AES256-SHA',
  'RSA-3DES-EDE-CBC-SHA',
  'ECDHE-RSA-AES128-SHA',
  'ECDHE-RSA-AES256-SHA',
  'ECDHE-ECDSA-AES128-SHA',
  'ECDHE-ECDSA-AES256-SHA',
  'DHE-RSA-AES128-SHA',
  'DHE-RSA-AES256-SHA',
  'DHE-RSA-3DES-EDE-CBC-SHA',
];

type SuiteMap = {
  [key: string]: string[];
};

const versionCipherMap: SuiteMap = {
  'v1.1': tls11CipherSuites,
  'v1.2': tls12CipherSuites,
};

function shuffledSuites(inputArray: string[]) {
  const firstThree = inputArray.slice(0, 3);
  const rest = inputArray.slice(3);
  for (let i = rest.length - 1; i > 0; i--) {
    const j = Math.floor(Math.random() * (i + 1));
    [rest[i], rest[j]] = [rest[j], rest[i]];
  }
  return firstThree.concat(rest).join(':');
}

function provideTLSAndSuites() {
  const tlsVersions = Object.keys(versionCipherMap);
  const selectedVersion = tlsVersions[Math.floor(Math.random() * tlsVersions.length)];
  const tlsVersionString = `--tls${selectedVersion}`;
  const shuffledSuitesString = shuffledSuites(versionCipherMap[selectedVersion]);
  return [tlsVersionString, shuffledSuitesString];
} This code provides the TLS version string for the request and selects and randomizes the cipher suites to prevent TLS fingerprinting. Final Curl Command We’ll fetch the TLS version and cipher suites for the request and embed them in the curl command: curl const [tlsVersionString, cipherSuitesString] = provideTLSAndSuites(); const [tlsVersionString, cipherSuitesString] = provideTLSAndSuites(); Here is the resulting curl command: curl curl --connect-timeout 30 --max-time 50 'url' \
 --proxy 'proxy_url' tlsVersionString --cipher cipherSuitesString \
 -H 'accept: application/json, text/javascript, */*; q=0.01' \
 -H 'accept-language: en-GB,en-US;q=0.9,en;q=0.8' \
 -H 'cookie: authCookie' \
 -H 'sec-ch-ua: "Not_A Brand";v="8", "Chromium";v="120", "Google Chrome";v="120"' \
 -H 'sec-ch-ua-bitness: "64"' \
 -H 'sec-ch-ua-full-version: "120.0.6099.234"' \
 -H 'sec-ch-ua-full-version-list: "Not_A Brand";v="8.0.0.0", "Chromium";v="120.0.6099.234", "Google Chrome";v="120.0.6099.234"' \
 -H 'sec-fetch-dest: empty' \
 -H 'sec-fetch-mode: cors' \
 -H 'sec-fetch-site: same-origin' \
 -H 'user-agent: userAgent' \
 -H 'x-requested-with: XMLHttpRequest' curl --connect-timeout 30 --max-time 50 'url' \
 --proxy 'proxy_url' tlsVersionString --cipher cipherSuitesString \
 -H 'accept: application/json, text/javascript, */*; q=0.01' \
 -H 'accept-language: en-GB,en-US;q=0.9,en;q=0.8' \
 -H 'cookie: authCookie' \
 -H 'sec-ch-ua: "Not_A Brand";v="8", "Chromium";v="120", "Google Chrome";v="120"' \
 -H 'sec-ch-ua-bitness: "64"' \
 -H 'sec-ch-ua-full-version: "120.0.6099.234"' \
 -H 'sec-ch-ua-full-version-list: "Not_A Brand";v="8.0.0.0", "Chromium";v="120.0.6099.234", "Google Chrome";v="120.0.6099.234"' \
 -H 'sec-fetch-dest: empty' \
 -H 'sec-fetch-mode: cors' \
 -H 'sec-fetch-site: same-origin' \
 -H 'user-agent: userAgent' \
 -H 'x-requested-with: XMLHttpRequest' Conclusion Hopefully, this will allow your request to make it through the firewall. If this helped you or you enjoyed the content, don’t forget to clap and follow for more such content. Happy scraping!

Defeating TLS Fingerprinting: Bypassing Firewall Protection for HTTPS Requests

Overview Digital fingerprinting is a technique used to identify users across different websites based on their unique device and browser characteristics. These characteristics - fingerprint parameters, can include various software, hardware (CPU, RAM, GPU, media devices - cameras, mics, speakers), location, time zone, IP, screen size/resolution, browser/OS languages, network, internet provider-related and other attributes. The combination of these parameters creates a unique identifier - fingerprint, that can be used to track a user's online activity. Fingerprints play a crucial role in online security, enabling services to identify and authenticate unique users. They also make it possible for users to trick such systems to stay anonymous online. However, if you can manipulate your fingerprints, you can run tens or hundreds or more different accounts to pretend that they are unique, authentic users. While this may sound cool, it has serious implications as it can make it possible to create an army of bots that can spread spam and fakes all over the internet, potentially resulting in fraudulent actions. Note: Obviously, I won’t discuss here how you can do "bad" things; you must always be careful, stay away from the “dark side,” and avoid committing illegal actions. This article is about the technology behind this, so use the information wisely. This is the first (and maybe not the last) article in this domain, so it will be more like an overview. The content offers an intro to the topic. There are dozens of tools and ways to detect, collect, and spoof fingerprints, and there are many different parameters and technologies that can reveal or hide your real identity or the fact that you're spoofing fingerprints (meaning you're not an authentic user). Parameters in Digital Fingerprints Let's consider some of them, some examples of obvious, simple, or well-known params, some that are rarer, less-know, and difficult to spoof. this string provides information about the user's browser, operating system, and device. User-Agent: it reveals the user's network and geo-location. Services use IP addresses for security and to prevent malicious activities. If you use one account from different IPs or many different accounts from the same IP, some services may view this activity (in combination with other params) as suspicious and use some level of bot protection against you. This can also be triggered if you use IPs that are already used or in some kind of banned list (proxies). IP: Information about installed plugins and extensions can be used to create a unique fingerprint - it helps in identifying users based on their browser's additional functionalities. Browser Plugins and Extensions: different users often have different display characteristics. Screen Resolution and Color Depth: they are important factors in fingerprinting - if you have random values, e.g., Japan timezone Norwegian language this is a bit unusual. Timezone and Language Settings: it involves rendering hidden graphics in the user's browser to gather information about the graphics hardware. Quite a tricky one because if you spoof your hardware info, then you can’t have the proper canvas value - you can’t render it on your different real hardware. Canvas: it exploits the unique capabilities and limitations of the user's graphics hardware in rendering 3D graphics - additional information about the user's device. WebGL Fingerprinting: the list of fonts installed on a user's system can be used as a fingerprinting parameter. This information is accessible through JavaScript. The list of fonts has to be realistic - so e.g. you can’t have Win fonts on macOS or you can’t have just 1 or 2 fonts, this is a very suspicious indicator. Fonts: it allows websites to determine the device's battery level and charging status, the combination of battery attributes can be used for fingerprinting. Battery status API: websites can use the web audio API to generate unique audio fingerprints by analyzing the audio processing characteristics of the device. Audio Fingerprinting: This information includes details about the software and hardware components of the device, such as the graphics card, network adapter, and operating system. This adds to the uniqueness of the fingerprint. Hardware Concurrency: This information includes details about the network connection, such as the IP address, ISP, and DNS server. Network Information: some users may use some software that listens to specific ports which also might be useful for some specific cases. Open and used ports: Navigating the Web incognito Digital fingerprint manipulation is a difficult task that demands a proactive approach to avoid detection. The are lots of strategies and tools to hide your identity (fingerprints) or pretend that you are someone else. Use privacy-focused browsers Opting for privacy-focused browsers, such as Brave, Ghostery, Tor, Octo Browser, or Vivaldi with enhanced privacy settings, provides a fundamental defense against common fingerprinting techniques. These browsers prioritize user privacy and incorporate features designed to hide your real fingerprints, making it harder to track your activities. For example, the Tor Browser, grounded in principles of anonymity, routes internet traffic through the Tor network. This strategic routing obscures the user's identity by bouncing connections through a series of volunteer-operated servers, enhancing overall online anonymity. Browser Extensions Privacy-centric browser extensions, including AdBlock, uBlock Origin, Privacy Badger, or CanvasBlocker, could be active defenses against tracking scripts, cookies, and fingerprinting attempts. These tools operate in the background, protecting and preserving user anonymity. VPN and Proxies The use of VPNs or proxy services adds an additional layer of protection by masking the user's real IPs (location, timezone, language, etc) and encrypting internet traffic. This is not only used to change your fingerprints but also to provide a more anonymous online presence and a higher security level. Motivations behind fingerprint spoofing Understanding the Whys Individuals concerned about personal privacy and the growing digital surveillance often spoof fingerprints to shield themselves from relentless online tracking. Privacy: Spoofing fingerprints prove useful for circumventing geo-restrictions, allowing users to access content restricted to specific regions. VPN and proxy services play a crucial role here by not only hiding the user's identity but also providing access to servers in different geographic locations. Geo-restriction: Avoiding targeted advertising and online profiling serves as a compelling motive for individuals to manipulate their digital fingerprints. Some tools can block 3rd-party tracking scripts and cookies, disrupting the profiling process. Ads: Web scraping, multi-accounting, e-commerce, bounty&airdrop, bonus-hunting, social networks bots, and affiliate marketing are often sources of income or bases for medium businesses. Digital agencies, individuals, and influencers engaged in these activities may require fingerprint spoofing to navigate through complicated bot detection systems. Avoiding detection becomes vital for some services, ensuring that legitimate activities are not mistakenly flagged or restricted (though quite often, such activities and accounts are not really legit but pretend to be so). Some of the most popular tools are Multilogin, X-Browser, Octo Browser, AdsPower, Incogniton, Scrapy, Surfsky, Web Scraper.io, ScrappingBee, etc. Strategic needs in the digital arena, bot detection, and protection systems: Collecting Users' Fingerprints Websites use JS (obviously) to harvest details about users to construct comprehensive digital portraits. This involves probing screen resolution, device orientation, mouse movements, keystroke dynamics, etc. Sophisticated fingerprinting scripts enumerate a wide range of browser and hardware attributes. JavaScript: Persistent cookies and data stored in local storage work to track users across sessions and platforms. Techniques include leveraging browser cookies and storing unique identifiers for user tracking. Depending on your purposes you need to know when you need an "empty" browser with unique fingerprints or when you need to trick a service showing that you have particular cookies and the same fingerprints. Cookies and local storage: The inclusion of 3rd-party scripts for analytics and advertising embeds invisible trackers often used to adjust systems behavior for a specific user based on collected fingerprints. You can use the aforementioned tools and approaches to selectively block or trick such scripts to get the needed behavior. 3rd-party scripts: Understanding Your Online Identity Panopticlick (EFF), Pixelscan, deviceinfo.me, and BrowserLeaks show your browser's fingerprint parameters such as User-Agent, canvas fingerprinting, fonts, etc. These checkers provide insights into the uniqueness and stability of your digital fingerprint across different browsing sessions. Online checkers: they allow users to inspect network requests, cookies, and other fingerprinting parameters, fostering a deeper understanding of their digital imprint and how websites detect bots and unique legit users and collect fingerprints. Examining the Network and Application tabs in browsers provides a real-time view of the data exchanged between the browser, websites, and servers. Browser developer tools: Here are some examples of how you can get some info about users' fingerprints in the browser. GEO: navigator.geolocation.getCurrentPosition(function(position) { var userLocation = position.coords; }); Microphone: navigator.mediaDevices.enumerateDevices() .then(function(devices) { var microphones = devices.filter(device => device.kind === 'audioinput'); }); Camera: navigator.mediaDevices.enumerateDevices() .then(function(devices) { var cameras = devices.filter(device => device.kind === 'videoinput'); }); Speakers: navigator.mediaDevices.enumerateDevices() .then(function(devices) { var speakers = devices.filter(device => device.kind === 'audiooutput'); }); Audio: var audioContext = new (window.AudioContext || window.webkitAudioContext)(); var oscillator = audioContext.createOscillator(); var analyser = audioContext.createAnalyser(); oscillator.connect(analyser); analyser.connect(audioContext.destination); GPU: var canvas = document.createElement('canvas'); var gl = canvas.getContext('webgl') || canvas.getContext('experimental-webgl'); var renderer = gl.getParameter(gl.RENDERER); console.log(renderer); Fonts: var fonts = []; var fontList = document.fonts.forEach(function(font) { fonts.push(font.family); }); console.log(fonts); Canvas (hashing): var canvas = document.createElement('canvas'); var context = canvas.getContext('2d'); var dataURL = canvas.toDataURL(); var canvasHash = MurmurHash3(dataURL); console.log(canvasHash); RAM and CPU: function getCPUInfo() { return navigator.hardwareConcurrency || 0; } function getDeviceMemory() { return navigator.deviceMemory || 0; } var cpuInfo = getCPUInfo(); var deviceMemory = getDeviceMemory(); console.log(`CPU Cores: ${cpuInfo}`); console.log(`Device Memory (GB): ${deviceMemory}`); Bot Detection Systems: Defending Against Automation A couple of examples of such systems: It is a global leader in bot detection and mitigation. It provides a proactive and precise way to verify that legitimate human users are accessing your website, mobile app, and APIs. Distil Networks uses a combination of techniques to detect bots. It studies a range of variables such as cursor movement, click patterns, and web browsing patterns across other websites. It uses , which checks all visitors against hundreds of different characteristics, focusing on their behavior. Distil Networks defends against web scraping, competitive data mining, account takeovers, transaction fraud, unauthorized vulnerability scans, spam, click fraud, denial of service, and API abuse. It can automatically block 99.9% of malicious traffic without impacting legitimate users. It also offers Distil Bot Defense for Web and API, which protect your website and API servers respectively. Distil Networks: Bot Detection: device fingerprints and Are You a Human technology It actively pulls additional data from the browser to identify devices with precision. When a browser request comes in, Distil interrogates the headers to see if the visitor is lying about their identity. This unique identifier can be used to fully or partially identify individual devices even when cookies cannot be read or stored in the browser, the client IP address is hidden, or one switches to another browser on the same device. It uses machine learning to detect biometric patterns in mouse activity and scrolling, as well as looking at browsers, devices, and other factors for clues. It can detect many simple or medium complexity bots that run JS from a web page. Resource Protection: It's a comprehensive cybersecurity platform that integrates fingerprint-based identification to discern legitimate users from potential threats. Utilizing advanced behavioral analysis and anomaly detection, Imperva builds profiles based on a combination of factors, including IP reputation, user-agent, different fingerprint params, and behavioral characteristics. This allows the system to detect suspicious activities indicative of bot traffic and mitigate potential threats effectively. Imperva: It's a prominent content delivery and cloud service provider that incorporates robust bot detection mechanisms within its security offerings. Leveraging a combination of fingerprint-based detection, behavior analysis, and machine learning, Akamai identifies and mitigates various bots. Akamai's global network allows for real-time threat intelligence, enabling proactive defense against evolving bot tactics. Akamai: It's a widely used content delivery network and security service that employs a multifaceted approach to bot detection. Cloudflare distinguishes between human users and bots by analyzing parameters such as IP reputation, user-agent characteristics, and behavioral patterns. The platform also utilizes threat intelligence and community-driven insights to stay ahead in the battle against emerging bot threats. Cloudflare uses to profile SSL/TLS clients and block potential bot requests. It also uses to apply rules and route traffic based on HTTP request information. is used to identify specific files and prevent data loss. These techniques help in detecting not real users. Cloudflare: JA3 Fingerprinting HTTP Filtering Data Fingerprinting Role of Bot Detection Systems: They play a pivotal role in enhancing bot detection capabilities. With servers strategically positioned around the globe, they can leverage geographical insights and real-time threat intelligence to identify and mitigate bot traffic effectively. The platform's WAF and bot managers contribute to a comprehensive defense against automated "users.” In this ongoing cat-and-mouse game, Bot Detection Systems continue to evolve, leveraging advanced technologies to stay ahead of fingerprint spoofing tech. The collaboration between security providers, businesses, and the wider online community remains crucial in defense against the tactics of automated threats but also, in a way, used against users' privacy, anonymity, and experience online, bringing intrusive ads and sophisticated user tracking tools that are only beneficial for businesses not for users. However, privacy protection, fingerprint hiding and spoofing, and ads and trackers blocking tools are also developing to evade even the most sophisticated user tracking systems. Websites' additional verification steps If a fingerprint appears suspicious, such as frequent changes or unusual combinations of data points, the website may trigger additional verification steps. Suspicious behavior: Websites may present users with captcha challenges to verify their identity. This involves solving puzzles or identifying distorted text, tasks that are difficult to perform for primitive bots. Captcha: Websites may request users to verify their mobile phone numbers to establish a stronger connection between a unique identity and a real-world entity. This can help prevent fraudulent activities. Mobile verification: the idea is the same as with mobile phones, but you need to submit your ID ID verification: Examples: Google doesn't require additional verification for new users with clear, consistent fingerprints. If a user's fingerprint changes frequently or resembles those used by known bots (or just already known for Google), Google may ask for a mobile phone number or ask to deal with a captcha. Financial websites often enforce stricter verification measures due to the sensitive nature of transactions. They may require users to provide additional personal information, verify their identity through secure channels, or pass more complex captcha challenges. Opinion: spoofing is better than hiding Hiding fingerprints can enhance privacy, security, and online anonymity, but spoofing them to appear as a unique, legitimate user offers even greater advantages. By blending in with the crowd and avoiding suspicion from systems that detect anti-detection tools, you can maintain credibility and trust. This approach allows you to enjoy the benefits of being perceived as a unique, legitimate user, minimizing the chance of facing additional protecting measures or obstacles. and, obviously, it helps in preventing leaking of your real data and identity. Also published . here

Intro to Digital Fingerprints: Understanding, Manipulating, and Defending Against Online Tracking

ডিজিটাল ফিঙ্গারপ্রিন্টের ভূমিকা: অনলাইন ট্র্যাকিংয়ের বিরুদ্ধে বোঝা, ম্যানিপুলেট এবং ডিফেন্ডিং

Einführung in digitale Fingerabdrücke: Online-Tracking verstehen, manipulieren und sich dagegen wehren

Introducción a las huellas digitales: comprensión, manipulación y defensa contra el seguimiento en línea

Introduction aux empreintes digitales : comprendre, manipuler et se défendre contre le suivi en ligne

डिजिटल फ़िंगरप्रिंट का परिचय: ऑनलाइन ट्रैकिंग को समझना, हेरफेर करना और उसके विरुद्ध बचाव करना

デジタル指紋の概要: オンライン追跡の理解、操作、および防御

디지털 지문 소개: 온라인 추적에 대한 이해, 조작 및 방어

Introdução às impressões digitais: compreensão, manipulação e defesa contra rastreamento on-line

Введение в цифровые отпечатки пальцев: понимание, манипулирование и защита от онлайн-слежения

Dijital Parmak İzlerine Giriş: Çevrimiçi Takibi Anlamak, Yönetmek ve Savunmak

Giới thiệu về Dấu vân tay kỹ thuật số: Hiểu, thao tác và bảo vệ chống lại việc theo dõi trực tuyến

数字指纹简介：理解、操纵和防御在线跟踪

is a biometric system that employs the distinctive patterns of raised ridges and recessed valleys on a person's fingertips to establish and verify their identity. It is one of the most popular and widely used biometric methods, as it offers several advantages, such as high accuracy, high convenience, and low cost. Fingerprint recognition However, it is also vulnerable to spoofing attacks, where an intruder tries to manipulate or fake a fingerprint to gain unauthorized access to a system or device. This begs the question: How real is the threat of fingerprint spoofing and how can we prevent it? This article will cover the following topics Fingerprint Spoofing: How real is the threat and how to prevent it Fingerprint Recognition Systems: How They Work and How to Use Them Safely Fingerprint Spoofing Attacks and Countermeasures The Role of the Industries Manufacturing Fingerprint Recognition Mechanisms The Role of the Government Agencies in Regards to Fingerprint Recognition Security Fingerprint Spoofing: How Real is the Threat and How Can It Be Prevented? Fingerprint spoofing is not a new phenomenon. It has been documented since the 19th century when criminals used various methods to alter or erase their fingerprints. In the 20th century, fingerprint spoofing became more sophisticated with the advent of new materials and techniques, such as latex, gelatin, silicone, wax, and glue. These materials can be used to create artificial fingerprints from molds, casts, or prints of genuine fingers. Alternatively, they can be used to coat a finger or a prosthetic device with a thin layer of fake skin that mimics the fingerprint pattern of another person. Fingerprint spoofing can be done for various purposes, such as impersonating someone else, bypassing security systems, or evading law enforcement. Some examples of fingerprint spoofing in real life include: According to BBC (2013) Publication, a Brazilian doctor used silicone fingers to fool a biometric attendance system and sign in for absent colleagues. According to Kim Zetter (2008) in a publication on Wired.com highlighted that a German hacker group claimed to have reproduced the fingerprint of the then-German Interior Minister from a glass he used at a press conference. According to CNBC (2018) Publication, a team of researchers from Michigan State University and New York University created fake fingerprints that could match multiple real fingerprints stored in a database. In the 2022 Times of India publication, 4 were arrested for cloning fingerprints in Aadhaar payments fraud. According to Biometricupdate.com (2021) Publication, Arrests in Pakistan for fake fingerprint biometrics, NADRA breach alleged These examples show that fingerprint spoofing is not only possible but also relatively easy and cheap to perform. Moreover, fingerprint spoofing can be done without the knowledge or consent of the legitimate owner of the fingerprint. This poses a serious challenge for the security and reliability of fingerprint recognition systems. How can we prevent fingerprint spoofing? There are several measures that can be taken to prevent or detect fingerprint spoofing. Some of them are: Using high-quality sensors that can capture the fine details and characteristics of fingerprints, such as pores, sweat glands, and blood vessels. Using liveness detection techniques that can distinguish between live and fake fingers based on physiological or behavioral features, such as temperature, pulse, blood pressure, skin texture, color, moisture, elasticity, response time, and movement. Using multi-factor authentication requires more than one biometric trait or other factors, such as passwords, PINs, tokens, or cards. Using biometric fusion combines the information from multiple biometric sensors or modalities to enhance the accuracy and robustness of recognition. Using anti-spoofing algorithms that can detect anomalies or inconsistencies in the fingerprint images or features. Educating users about the risks and best practices of using fingerprint recognition systems. Fingerprint Recognition Systems: How They Work and How to Use Them Safely Fingerprint recognition systems consist of four main components: a sensor, a feature extractor, a matcher, and a database. The basic steps involved in fingerprint recognition are: The user registers their fingerprint by placing their finger on the sensor. The sensor captures an image of the fingerprint and sends it to the feature extractor. The feature extractor extracts the salient features of the fingerprint, such as minutiae points (the endings and bifurcations of the ridges), ridge orientation, frequency, and shape. The features are then stored in a template in the database, along with the user's identity information. Enrollment: The user presents their fingerprint again to the sensor for verification. The sensor captures an image of the fingerprint and sends it to the feature extractor. The feature extractor extracts the features of the fingerprint and compares them with the template stored in the database. The matcher then calculates a similarity score between the two sets of features and determines whether they match or not. If the score is above a predefined threshold, the user is authenticated; otherwise, they are rejected. Verification: : The user presents their fingerprint to the sensor for identification. The sensor captures an image of the fingerprint and sends it to the feature extractor. The feature extractor extracts the features of the fingerprint and compares them with all the templates stored in the database. The matcher then ranks the templates according to their similarity scores with the input fingerprint and returns the identity of the most similar one. Identification How to use fingerprint recognition systems safely Fingerprint recognition systems are not foolproof. They may be susceptible to a range of attack types, including: Spoofing: An attacker creates a fake fingerprint from a mold, cast, or print of a genuine finger and uses it to deceive the sensor. Alteration: An attacker modifies their own fingerprint by cutting, burning, or grafting skin to change its pattern. Obfuscation: An attacker covers their own fingerprint with dirt, oil, or other substances to reduce its quality or clarity. Denial: An attacker damages or destroys the sensor or the database to prevent legitimate users from accessing the system. Replay: An attacker captures and replays a fingerprint image or signal to the sensor or the system.
Notably, to prevent or mitigate these attacks, users should follow some best practices when using fingerprint recognition systems, such as: Keep your fingers clean and dry before touching the sensor. Do not share your fingerprints with anyone or leave them on surfaces that can be easily accessed by others. Use additional factors of authentication, such as passwords, PINs, tokens, or cards, along with your fingerprints. Choose a reliable and secure system that has anti-spoofing and liveness detection capabilities. Report any suspicious or abnormal activities or incidents related to your fingerprints or the system. Fingerprint Spoofing Attacks and Countermeasures Fingerprint spoofing attacks are a type of presentation attack, where an adversary tries to fool a fingerprint recognition system by presenting a fake or altered fingerprint. These attacks pose a serious threat to the security and privacy of biometric systems, as they can compromise the identity and access of legitimate users. In this article, we survey the state-of-the-art methods and techniques for fingerprint spoofing attacks and countermeasures and discuss the challenges and future directions in this field. Fingerprint spoofing attacks can be classified into two categories: and . Direct Indirect involve the fabrication and presentation of a fake fingerprint that mimics the appearance and features of a genuine one. Direct attacks involve the modification or obfuscation of a genuine fingerprint to alter its recognition outcome. Indirect attacks can be further divided into three types: 2D, 3D, and cadaver. 2D attacks use flat materials, such as paper, film, or tape, to print or transfer a fingerprint image onto a surface that can be placed on the sensor. 3D attacks use materials that can create a relief or a mold of a fingerprint, such as latex, gelatin, silicone, wax, or glue. Cadaver attacks use real human fingers that are detached from corpses or amputated from living persons. Direct attacks can be further divided into four types: alteration, obscuration, denial, and replay. Indirect attacks Alteration attacks use physical or chemical methods to change the shape or pattern of a fingerprint, such as cutting, burning, grafting, or bleaching. Obscuration attacks use substances or objects to cover or degrade the quality of a fingerprint, such as dirt, oil, gloves, or rings. Denial attacks use methods to damage or destroy the sensor or the system components, such as smashing, tampering, or hacking. Replay attacks use methods to capture and replay a fingerprint image or signal to the sensor or the system, such as eavesdropping, recording, or injecting. Fingerprint spoofing countermeasures are techniques that aim to detect and prevent fingerprint spoofing attacks. They can be classified into two categories: and . hardware-based software-based use additional sensors or devices to capture additional information about the fingerprint or the finger, such as temperature, pulse, blood pressure, skin texture, color, moisture, elasticity, response time, and movement. Software-based countermeasures use algorithms or models to analyze the fingerprint image or signal and extract features or patterns that can distinguish between genuine and fake fingerprints. Hardware-based countermeasures Hardware-based countermeasures have the advantage of being more robust and reliable than software-based ones, as they can capture physiological or behavioral characteristics that are difficult to fake or modify. However, they also have some drawbacks, such as being more expensive, complex, intrusive, and user-unfriendly than software-based ones. have the advantage of being more flexible and adaptable than hardware-based ones, as they can be implemented on existing sensors and systems without requiring additional hardware. However, they also have some drawbacks, such as being more vulnerable to noise, variability, and spoofing techniques than hardware-based ones. Software-based countermeasures : Some examples of hardware-based countermeasures are Thermal sensors that measure the heat emitted by the finger. Optical sensors that capture the subsurface structure of the finger. Capacitive sensors that measure the electric charge of the finger. Pressure sensors that measure the force applied by the finger. Liveness detection devices that verify the presence of vital signs in the finger. Some examples of software-based countermeasures are: Texture analysis methods that extract features from the fingerprint image based on its gray-level distribution. Frequency analysis methods that extract features from the fingerprint image based on its spectral properties. Minutiae analysis methods that extract features from the fingerprint image based on its ridge endings and bifurcations. Machine learning methods that use classifiers or neural networks to learn patterns from genuine and fake fingerprints. Anti-spoofing algorithms that detect anomalies or inconsistencies in the fingerprint image or signal. Fingerprint spoofing attacks and countermeasures are an active and evolving research area in biometric security. Some of the challenges and future directions in this field are: Developing more realistic and diverse spoofing materials and techniques that can challenge existing countermeasures. Developing more robust and accurate countermeasures that can cope with different types and qualities of fingerprints and sensors. Developing more standardized and comprehensive datasets and protocols for evaluating fingerprint spoofing attacks and countermeasures. Developing more user-friendly and privacy-preserving countermeasures that can balance security and convenience for users. Developing more integrated and holistic countermeasures that can combine multiple hardware and software techniques for enhanced protection. The Role of the Industries Manufacturing Fingerprint Recognition Mechanisms The industries manufacturing fingerprint recognition mechanisms should play an important role in guarding against fingerprint spoofing attacks. They should: Follow the standards and guidelines for fingerprint recognition systems developed by the National Institute of Standards and Technology (NIST) and other relevant organizations. These standards and guidelines specify the formats, protocols, and performance requirements for exchanging and processing fingerprint data. . Implement the best practices and recommendations for fingerprint recognition systems provided by NIST and other experts. These best practices and recommendations cover various aspects of fingerprint recognition systems, such as sensor design, image quality, feature extraction, matching algorithms, liveness detection, anti-spoofing techniques, and system security. . Conduct regular testing and evaluation of their fingerprint recognition mechanisms using the tools and methods developed by NIST and other entities. These tools and methods enable the industries to measure the accuracy, reliability, interoperability, and robustness of their fingerprint recognition mechanisms against various types of spoofing attacks. . Collaborate with other stakeholders, such as government agencies, academic institutions, and user groups, to share information, experiences, and feedback on fingerprint recognition systems. This collaboration can help the industries to identify the emerging trends, challenges, and opportunities in fingerprint recognition technologies and applications. . The Role of the Government Agencies in Regards to Fingerprint Recognition Security The role of the government agencies in regards to fingerprint recognition security is to: Develop and implement policies and regulations that govern the collection, storage, sharing, and use of fingerprint data for various purposes, such as law enforcement, immigration, national security, and civil services. These policies and regulations should balance the benefits and risks of fingerprint recognition systems, and protect the rights and privacy of individuals. Establish and maintain standards and guidelines for fingerprint recognition systems, such as the formats, protocols, and performance requirements for exchanging and processing fingerprint data. These standards and guidelines should ensure the interoperability, accuracy, reliability, and robustness of fingerprint recognition systems across different domains and platforms. Support and fund research and development activities that aim to improve the quality, efficiency, and security of fingerprint recognition systems. These activities should include the evaluation and testing of existing and emerging technologies, methods, and techniques for fingerprint capture, extraction, matching, liveness detection, anti-spoofing, and biometric fusion. , Fingerprint recognition is a powerful and convenient technology that can provide high levels of security and convenience for various applications. However, it is not immune to spoofing attacks that can compromise its effectiveness and trustworthiness. Therefore, it is important to use it wisely and responsibly and to implement appropriate countermeasures that can prevent or detect fingerprint spoofing and ensure the integrity and privacy of biometric data. In Conclusion Moreover, it is essential to have a collaborative and coordinated effort among the industries manufacturing fingerprint recognition mechanisms, the government agencies regulating and supporting fingerprint recognition systems, and the users and stakeholders benefiting from fingerprint recognition systems. By doing so, we can enhance the security and reliability of fingerprint recognition systems and protect the identity and access of legitimate users. Featured image by Geralt on Pixabay

Fingerprint Recognition Security: A Concise Guide On The Need To Know

A device fingerprint - or device fingerprinting - is a method used to identify a device with a combination of attributes provided by the device itself, via its browser and device configuration. The attributes collected as data to build the device fingerprint depend on the solution used to build it, but typically the most common are: operating system screen size and resolution user-agent system language and system country device orientation battery level installed fonts and installed plugins system uptime IP address HTTP request headers Since most of these parameters are read from the browser settings, we can also use the term “browser fingerprinting” with the same connotation. If you want to test which machine features are leaked from your browser just by browsing a web page, , simply with a Javascript executed on the server. you can use this online test to check with your eyes Consider also that most of the common anti-bot solutions use this basic information and enrich them with more complex test results, like Canvas and WebGL fingerprinting, to add even more details to these fingerprints. : the more pieces I add to the fingerprint, the more granular it becomes (because it’s less likely for two users to have the exact same device configuration). In this way, I can abstract a small niche of users with one fingerprint and track their behavior, without using cookies. The point is And this is key in these times when cookies are under the scrutiny of GDPR, CCPA, and other internet regulations and users are getting more aware of them, deciding to opt out or wipe them from their machines. But it’s not only a matter of marketing, but also anti-fraud and anti-bot in general are involved in developing this kind of technologies. In fact, detecting fingerprints that contain some incompatible data or outliers in configurations, can raise some red flags in the traffic. How a device fingerprint is collected? As we have seen, a fingerprint is a collection of single pieces of information collected when a device connects to a server. Depending on the method used for collecting this information, we can divide them by and fingerprinting techniques. active passive is when a server interacts with the device, using a challenge that the browser needs to solve. An example can be the Canvas fingerprinting or WebGL fingerprinting technique. In both cases, an image with a text overlay is rendered off-screen. On different hardware, this image and its result hash string are rendered differently, so we have different fingerprints for different hardware. Active fingerprinting is when the server simply gathers the information passed by the browser and the different HTTP connection layers: IP address, request headers, user agent, screen resolution, operating system, and so on. Passive fingerprinting Modern anti-bot software combines both these families of techniques and integrates them with behavioral analysis and AI to detect incongruences in the settings of the device and the scraper. If you’re curious about the number and the type of information that can be gathered via your web browser, you can have a look at deviceinfo.me, an online test where you can discover all these details. As you can see from the following images, the description of your device is quite accurate. Fingerprinting and privacy concerns As we mentioned before, fingerprinting techniques are gaining traction not only in the anti-bot industry but also in the marketing one, since more and more people are concerned about cookie usage by websites. Since it’s possible to accept or decline the cookie usage of websites and, since they are stored in the user's device, they can also be deleted, their efficacy for marketing purposes is declining. But if you’re using a fingerprinting tool in your online marketing solution, maybe you have a less granular detail than the single users’ cookie collection but there’s no way to opt out for them. So you can create very granular clusters of customers, like “people from city X using the latest Mac laptop model, with a second screen, browsing via Chrome v. 113 with Y extension installed, and connecting with ISP Z”. It is such a detailed description that the European think tank about online privacy called , expressed its opinion about fingerprinting and European data protection laws. Article 29 Data Protection working party To make a long story short: collecting all these pieces of information from the browser to create a unique fingerprint makes the ePrivacy directive applicable to this technology. This implies that visitors of a website, just like what happens for cookies, should be informed if there are any fingerprinting techniques used on the website, unless they are meant only to make the website work correctly. How to mask your fingerprint Having a look at the website, we can notice the several layers of information gathered to create a fingerprint. Connection layer A TLS fingerprint is created using the handshake packets that the client and server exchange before establishing an HTTP connection. It’s quite a common technique used by major anti-bot providers, . as we can see from this Cloudflare article To avoid raising red flags when scraping, you can use real browsers with Playwright or Selenium, which will use ciphers not in the blacklist, or change ciphers in your Scrapy project’s settings. Of course, the server knows also the IP address of the device that is connecting and can derive from it several additional info, like your country, state, ISP, and so on. We can change all these details by using a proxy provider and depending on your needs, you can use datacenter, residential or mobile IPs. Browser layer fingerprint Most of the other information used to fingerprint you are coming from your browser settings and how the browser reacts to Canvas and WebGL active fingerprinting. Typically, if you’re trying to scrape a website protected by a modern anti-bot solution, you cannot use solutions like Scrapy but you’ll need to use a webdriver or a real browser to bypass the protection. In these cases, providing a plausible machine setup with no discrepancies in the settings is key to creating a fingerprint that seems as legit as possible. As an example, using Playwright and Chrome in headless mode should trigger some red flags, since it’s easy to detect, as we can see from screenshots below. Another example is connections from a server machine won’t show any microphone and camera. But if you use anti-detect browsers, which allow you to create custom profiles that mimic different hardware and OS setups, you can send more plausible information to the server, creating a more legit fingerprint. Final remarks We have seen how much information is transmitted together with a simple browser connection to a website and how they can be used to both track users’ behavior and detect bots. While all these fingerprinting techniques can raise some concerns about privacy, especially in countries where this information can be used to limit private freedom, we luckily have several tools in our toolbelt to use to mask our real online traces, and some of them can be used also for our web scraping projects. Also published here.

Understanding the Fundamentals of Device Fingerprinting 

Once out of reach, parts of our world are now awaiting exploration through an interdisciplinary approach in AI development. The future is now. For example, researchers at the University of Jyväskylä are creating an interdisciplinary bridge between dance movements and machine learning technologies. It appears that dancing might be on its way to guide the IT world towards a better understanding of communication, leaving a noticeable mark on the future of AI algorithm development. How do Argentine Tango’s interactive characteristics contribute to machine learning development? Could it help IT professionals transcribe dancing movements into more efficient lines of code?  Learning how to reflect on can be a helpful step in humanizing robotics. tango-based enactivism The undeniable power of the individual mark In conducted , a hypothesis stated that an algorithm can categorize different dancing genres (such as jazz, hip-hop, or blues) based on dancers’ movements. As it turns out, the machines were “thinking” differently to humans, surprising researchers with unexpected outcomes. research to use machine learning and computational algorithms to distinguish and adequately categorize eight dancing genres based on motion capture data from participants dancing freely. The research’s primary goal was The outcome was that the machines recognized personal dancing styles as an individual fingerprint, instead of specifying dance genres based on previously provided data. The researchers assumed that the machine learning algorithm would easily recognize different dancing styles, as each of them has its own unique characteristics, such as tempo, steps, and overall choreography. In order to verify the hypothesis, researchers assembled a group of participants whose movements were recorded using an optical motion capture system. It followed the three-dimensional positions of the dancers based on 21 reactive markers attached to each participant’s body. The data was then transcripted into a set of 20 secondary markers for further accuracy, and marker locations were adjusted accordingly (except the S, and T locations, which remained the same). were placed as shown in . Primary markers picture A can be viewed in . Secondary markers picture B Photo Credit: Emily Carlson , University of Jyväskylä | JYU · Department of Music Unlike researchers expected, the output they received identified the more personal ID of dancing rather than previously defined genres. Although their initial assumptions were different, these results can provide a whole new world of perspectives for research regarding unique dancing styles and machine learning. Nowadays, interdisciplinary connections are necessary to create a progress-oriented environment with an awareness of how interaction is crucial for societal development. Machine learning is well on its way to recognizing members of society through their personal movements. Enactivism: Interaction On a Deeper Level So, how can we connect those research results with people’s daily lives and Argentinian tango? Without further ado, the answer lies under the definition of enactivism. We can understand it as evoking cognitive functions through sensomotoric activity. In other words, it is based on interacting with an environment through receiving information and incorporating it into sense-making processes. What’s most important about this type of cognitive system is that it doesn’t just absorb specific details from the surroundings and incorporate them into a brain representation without any further analysis. It focuses on integrating both passive and active signals from the environment. Tango can be defined as consciously coordinated interaction with a partner. Connections in Tango are all about determining how to create . This is followed by understanding the partner’s mood and being reactive to or foreseeing movements in the environment, unifying the whole experience into one gracious flow of interaction. How does this all align with Argentinian Tango? specific energy between tangueros and tangueras Argentine Tango, in its , means an improvised but also synchronized structure of movements. It involves the engagement of two individuals on their path to learning coordinated interactions through mutual understanding. enactive interpretation As part of a complex cognitive system for years, the embodiment is being used to develop machine learning algorithms and being steered towards an enactive approach. Now, it is also interpreted as an inseparable part of Tango in terms of participatory sense-making, mutual incorporation, and consensually coordinated action. Enactivism combines both Tango and machine learning to progress on the study of the complexity of human-robot interaction. IT professionals should not reduce machines to receivers; they have to interact with their environments to develop algorithms. The attempts to map human cognition should incorporate the enactive approach, developing machines into sensitive and receptive algorithms. IT professionals: Tango your way to algorithms When designing an AI algorithm, it’s essential to set up what kind of outcome you expect primarily. Suppose the robotics industry aims to teach machines how to interact with their environment. In that case, it might be a valuable experience for IT professionals to explore enactivism through different sources, such as Argentinian Tango. How can IT professionals transfer Tango to their work? Through exploring the real-world experiences that they wish to include in machine learning development. The question is how adequately scientists and engineers can imitate people’s emotions, improvisational abilities, level of coordination, and sense-making processes. Researchers combine mental and physical involvement with the environment to understand the body’s movement through dance. Robotics is taking a new direction, in which machines are designed to interact with the environment through developing sense-making processes based on enactivism. I suppose the : are robots and machines meant to be rigid and emotionless or are they possible to develop new interactions through incorporating dancing techniques? The future will tell. question remains By Anita Flejter , Argentine tango expert and Co-Founder of Ultimate Tango .

A Unique Interaction: Learning the Code of Dancing Fingerprints

Stories, News & Mentions about FINGERPRINT

Fingerprint’s Stories on HackerNoon