Sense

sense

Author:
(1) Thierry Boy de la Tour, Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG 38000 Grenoble, France. Table of Links Abstract and 1 Introduction 2 Basic Definitions and Notations 2.1 Sets 2.2 Sequences 2.3 Signatures and Algebras and 2.4 Categories 3 Monographs and their Morphisms 4 Limits and Colimits 5 Drawing Monographs 6 Graph Structures and Typed Monographs 7 Submonographs and Partial Morphisms 8 Algebraic Transformations of Monographs 9 Attributed Typed Monographs 10 Conclusion and References 9 Attributed Typed Monographs The notion of E-graph has been designed in [2] in order to obtain an adhesive category of graphs with attributed nodes and edges. This follows from a line of studies on Typed Attributed Graph Transformations, see [21, 22, 23]. The attributes are taken in a data type algebra and may be of different sorts (booleans, integers, strings, etc.). In the case of E-graphs only the nodes of sort values represent such attributes. But they are also typed by E-graphs, and in the type E-graphs each node of sort values represent a sort of the data type algebra. This should recall the constructions of Section 6 that we now use in order to generalize the notion of typed attributed graphs given in [2]. The idea is similarly to impose that the edges typed by a sort of a data type algebra are the elements of the corresponding carrier set. Theorem 9.4 generalizes[3] [2, Theorem 11.3] that establishes an isomorphism between the category of attributed E-graphs typed by an attributed E-graph AT G and the category of algebras of a signature denoted AGSIGpAT Gq. In particular Theorem 11.3 of [2] requires the hypothesis that AGSIGpAT Gq should be well-structured, which means that if there is an operator name of ST whose domain sort is s then s is not a sort of the data type signature Σ. Obviously this is equivalent to requiring that only nodes of T can be considered as sorts of Σ and is linked to the fact that only values nodes of E-graphs are supposed to hold attributes. Since we are not restricted to E-graphs there is no need to require that attributes should only be nodes. This has an interesting consequence: The approach adopted in [2, Definition 11.5] is to restrict the morphisms used in span rules to a class of monomorphisms that are extensions of Σ-isomorphisms to pΓ `Σq-homomorphisms. It is then possible to show [2, Theorem 11.11] that categories of typed attributed E-graphs are adhesive HLR categories (a notion that generalizes Definition 4.13, see [24]) w.r.t. this class of monomorphisms. A similar result holds on categories of ATMs. For the sake of simplicity, and since rule-based graph transformations are unlikely to modify attributes such as booleans, integers or strings (and if they do they should probably not be considered as graph transformations), we will only consider morphisms that leave the data type algebra unchanged, element by element. This leaves the possibility to transform the edges whose sort is in Γ but not in Σ. The proof that the categories ATMpT, Aq are adhesive will only be sketched below. The key point is the following lemma. Hence the property of stabilization characterizes the difference between morphisms in Monogrz\T and morphisms in ATM(T, A). Besides, it is well-known how pushouts and pullbacks in Monogrz\T can be constructed from those in Monogr, and we have seen that these can be constructed from those in Sets. Theorem 9.8. ATM(T, A) is adhesive. This result does not mean that all edges that are not attributes can be freely transformed. Their adjacencies to or from attributes may impose constraints that only few morphisms are able to satisfy. Example 9.9. Let Σ be the signature with no operation name and one sort s, and A be the Σ-algebra defined by As “ ta, bu. We consider the type monograph T “ tpe, sq,ps, equ. A monograph typed by T has any number (but at least one) of edges typed by e that must be adjacent either to a or b, and two edges typed by s, namely a and b, that must be adjacent to either the same edge x typed by e, which yields two classes of monographs (to which may be added any number of edges typed by e and adjacent to either a or b), or a and b are adjacent to y and z respectively, and we get four more classes: The function y, z → x is a morphism from these last two monographs to the two monographs above (respectively). There are no other morphisms between monographs from distinct classes. We therefore see that in the category ATM(T, A) it is possible to add or remove edges typed by e to which a or b are not adjacent, but there is no way to remove the edges y and z (because this would require a rule with a left morphism from an ATM without y and z to an ATM with y and z, and there is no such morphism), though they are not attributes. Besides, we see that this category has no initial object, no terminal object, no products nor coproducts. This paper is available on arxiv under CC BY 4.0 DEED license. [3] Our proof is also much shorter than the 6 pages taken by the corresponding result on attributed typed E-graphs. This is due partly to our use of AT (Definition 6.7) and of Theorem 6.8, but also to the simplicity of monographs compared to the 5 sorts and 6 operator names of E-graphs. Author: (1) Thierry Boy de la Tour, Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG 38000 Grenoble, France. Author: Author: (1) Thierry Boy de la Tour, Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG 38000 Grenoble, France. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Basic Definitions and Notations 2.1 Sets 2.1 Sets 2.2 Sequences 2.2 Sequences 2.3 Signatures and Algebras and 2.4 Categories 2.3 Signatures and Algebras and 2.4 Categories 3 Monographs and their Morphisms 3 Monographs and their Morphisms 4 Limits and Colimits 4 Limits and Colimits 5 Drawing Monographs 5 Drawing Monographs 6 Graph Structures and Typed Monographs 6 Graph Structures and Typed Monographs 7 Submonographs and Partial Morphisms 7 Submonographs and Partial Morphisms 8 Algebraic Transformations of Monographs 8 Algebraic Transformations of Monographs 9 Attributed Typed Monographs 9 Attributed Typed Monographs 10 Conclusion and References 10 Conclusion and References 9 Attributed Typed Monographs The notion of E-graph has been designed in [2] in order to obtain an adhesive category of graphs with attributed nodes and edges. This follows from a line of studies on Typed Attributed Graph Transformations, see [21, 22, 23]. The attributes are taken in a data type algebra and may be of different sorts (booleans, integers, strings, etc.). In the case of E-graphs only the nodes of sort values represent such attributes. But they are also typed by E-graphs, and in the type E-graphs each node of sort values represent a sort of the data type algebra. This should recall the constructions of Section 6 that we now use in order to generalize the notion of typed attributed graphs given in [2]. The idea is similarly to impose that the edges typed by a sort of a data type algebra are the elements of the corresponding carrier set. Theorem 9.4 generalizes[3] [2, Theorem 11.3] that establishes an isomorphism between the category of attributed E-graphs typed by an attributed E-graph AT G and the category of algebras of a signature denoted AGSIGpAT Gq. In particular Theorem 11.3 of [2] requires the hypothesis that AGSIGpAT Gq should be well-structured, which means that if there is an operator name of ST whose domain sort is s then s is not a sort of the data type signature Σ. Obviously this is equivalent to requiring that only nodes of T can be considered as sorts of Σ and is linked to the fact that only values nodes of E-graphs are supposed to hold attributes. Since we are not restricted to E-graphs there is no need to require that attributes should only be nodes. This has an interesting consequence: The approach adopted in [2, Definition 11.5] is to restrict the morphisms used in span rules to a class of monomorphisms that are extensions of Σ-isomorphisms to pΓ `Σq-homomorphisms. It is then possible to show [2, Theorem 11.11] that categories of typed attributed E-graphs are adhesive HLR categories (a notion that generalizes Definition 4.13, see [24]) w.r.t. this class of monomorphisms. A similar result holds on categories of ATMs. For the sake of simplicity, and since rule-based graph transformations are unlikely to modify attributes such as booleans, integers or strings (and if they do they should probably not be considered as graph transformations), we will only consider morphisms that leave the data type algebra unchanged, element by element. This leaves the possibility to transform the edges whose sort is in Γ but not in Σ. The proof that the categories ATMpT, Aq are adhesive will only be sketched below. The key point is the following lemma. Hence the property of stabilization characterizes the difference between morphisms in Monogrz \T and morphisms in ATM (T, A). Besides, it is well-known how pushouts and pullbacks in Monogrz \T can be constructed from those in Monogr , and we have seen that these can be constructed from those in Sets. Monogrz ATM Monogrz Monogr Theorem 9.8. ATM (T, A) is adhesive. Theorem 9.8. ATM This result does not mean that all edges that are not attributes can be freely transformed. Their adjacencies to or from attributes may impose constraints that only few morphisms are able to satisfy. Example 9.9. Let Σ be the signature with no operation name and one sort s, and A be the Σ-algebra defined by As “ ta, bu. We consider the type monograph T “ tpe, sq,ps, equ. A monograph typed by T has any number (but at least one) of edges typed by e that must be adjacent either to a or b, and two edges typed by s, namely a and b, that must be adjacent to either the same edge x typed by e, which yields two classes of monographs Example 9.9. Let Σ be the signature with no operation name and one sort s, and A be the Σ-algebra defined by As “ ta, bu. We consider the type monograph T “ tpe, sq,ps, equ. A monograph typed by T has any number (but at least one) of edges typed by e that must be adjacent either to a or b, and two edges typed by s, namely a and b, that must be adjacent to either the same edge x typed by e, which yields two classes of monographs (to which may be added any number of edges typed by e and adjacent to either a or b), or a and b are adjacent to y and z respectively, and we get four more classes: (to which may be added any number of edges typed by e and adjacent to either a or b), or a and b are adjacent to y and z respectively, and we get four more classes: The function y, z → x is a morphism from these last two monographs to the two monographs above (respectively). There are no other morphisms between monographs from distinct classes. We therefore see that in the category ATM(T, A) it is possible to add or remove edges typed by e to which a or b are not adjacent, but there is no way to remove the edges y and z (because this would require a rule with a left morphism from an ATM without y and z to an ATM with y and z, and there is no such morphism), though they are not attributes. The function y, z → x is a morphism from these last two monographs to the two monographs above (respectively). There are no other morphisms between monographs from distinct classes. We therefore see that in the category ATM(T, A) it is possible to add or remove edges typed by e to which a or b are not adjacent, but there is no way to remove the edges y and z (because this would require a rule with a left morphism from an ATM without y and z to an ATM with y and z, and there is no such morphism), though they are not attributes. Besides, we see that this category has no initial object, no terminal object, no products nor coproducts. Besides, we see that this category has no initial object, no terminal object, no products nor coproducts. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv [3] Our proof is also much shorter than the 6 pages taken by the corresponding result on attributed typed E-graphs. This is due partly to our use of AT (Definition 6.7) and of Theorem 6.8, but also to the simplicity of monographs compared to the 5 sorts and 6 operator names of E-graphs.

Making Sense of Attributed Typed Monographs

It always starts small. A CSV file here, some JSONs there. You write some good python code using Pandas, maybe some NumPy, and everything runs as expected. Fast. Simple. Beautiful. But then... the files grow. Queries slow down. You get the dreaded “Out of Memory” error. Now all of a sudden, you’re questioning whether to elevate your game. And for many of us, that is one word: Spark. But when is it really worth migrating from the old faithful python to Spark? When does scaling up help solve your problems—and when does it simply complicate everything further? Let’s figure that out. Python: The Single-Machine Workhorse For faster uses, Python has no rivals and allows you to manipulate small to medium datasets. If your entire dataset is comfortably loaded in your machine’s memory (RAM), Python + Pandas or Polars will usually deliver the fastest development time and cleanest code. And let’s be honest: many of us start here because it’s easy. Install Python. Install Pandas. Write a few lines. Done. Why stick with Python? Ease of use. Easy set up, great documentation and big ecosystem.


Readable code. Most of the workflows follow intuitively from Pandas, NumPy, and just Python built-ins.


Speed for small jobs. Python is fast for datasets that are a couple gigs or smaller.


Local development. No clusters, distributed systems, etc.


Fast prototyping. Prototype quickly and test concepts without waiting on infrastructure. 🔗 Pandas Documentation 🔗 Polars Documentation Common Python sweet spots: ETL scripts
Data cleaning
API data parsing
Exploratory Data Analysis (EDA)
Prototypes and notebooks
Ad hoc data tasks But... here's the catch: Python, and therefore Pandas is single-threaded and memory-bound. As soon as you start having your dataset bigger than your RAM, it is getting slow. Really slow. The first indication is usually the sound of your machine’s fan kicking into overdrive as if it were at the beginning of a flight. Then the lag creeps in. You might try chunking the data, or writing a for-loop to process files individually. You begin to sprinkle in “TODO” comments for edge case handling, telling yourself that you’ll “optimize this later.” At some point, Python literally just... gives up. Spark: The Distributed Heavy Lifter Spark was created for big data. We are talking terabytes or petabytes over several machines. The key to Spark’s performance is that it scales horizontally, allowing it to divide work across a cluster of servers and process chunks of your data at the same time. Spark doesn’t care if your laptop has 8GB of RAM, because it solves jobs that require hundreds of gigabytes, or even more. Why move to Spark? Large datasets that cannot be fitted in a single machine.


Processing in parallel both across CPUs and across servers.


Join multi-levels, group-bys or complex transformations in batch.


Built-in fault tolerance.


Strong integrations with cloud storage (S3, ADLS, GCS, etc.)


Reliable and predictable runs of production-grade batch pipelines. 🔗 Apache Spark Documentation 🔗 PySpark Documentation Spark shines when: You are joining several large datasets.
You have business-critical jobs that run daily (or hourly).
Your data is expanding exponentially and you realize that soon you’ll outgrow a single node.
The company already has a cluster, you just need to connect into an existing environment. But (and it’s a big but)… Spark comes with overhead. Setting up Spark and managing clusters, tuning performance, it’s not easy. For small jobs Spark can be slower than Python as the time taken to start the job, distribute the data and shuffle the output can be greater than the benefits provided by Parallelism. 🚩 5 Signs You Need to Scale Your Business Up Not convinced that you should make the switch to Spark? Here are some clear signs your current Python setup may be throwing in the towel: 1️⃣ Out-of-Memory Errors If your scripts are regularly crashing due to memory issues, and you’ve optimized your data types and chunked where possible already … then it might be your time. 2️⃣ Jobs Taking Too Long When a job that once took minutes now takes hours — or worse, does not complete at all — scaling out via distributed processing can help reclaim your time (and sanity). 3️⃣ Frequent Timeouts or Failures Pipelines that arbitrarily error out at resource boundaries are ops hell. These loads are much more graceful with distributed systems like spark. 4️⃣ You’re Splitting Work by Hand If you realize that you are writing loops to manually pick up the data in pieces, process it, and join it back together, that is literally what Spark does on its own—only with more effort and risk. 5️⃣ Companies Growing Rapidly Despite Their Architecture If your company is scaling quickly and data volumes are rising, switching before things break is preferable to switching after. 6️⃣ Your Hardware Scaling Isn’t Making a Difference Upgraded your machine from 16GB to 32GB of RAM, are you still struggling? Single-machine power can only be pushed so far. At times it’s not just about scaling up, but also scaling out. How to Choose: Python VS Spark? Here’s my own cheat sheet: Scenario

Use Python

Use Spark



Dataset < 5GB

✅

❌



Dataset > 10GB

🚩

✅



Quick prototype

✅

❌



Daily production ETL

Maybe

✅



Heavy joins & group-bys

❌

✅



Lots of small API calls

✅

❌



Cluster available

Maybe

✅



Just you on a laptop

✅

❌ When the Wrong Choice Gets Made I've also watched people switch to Spark much too soon. They heard "big data" and believed every CSV required a cluster. But clusters are expensive. Managing them takes time. And when Spark is abused for small datasets, the overhead can destroy performance. On the other end, I do hate seeing failed teams get stuck with complex, memory-heavy Pandas code bases that crash every day… when a small Spark job would’ve done the trick just fine. The moral? A tool should not be selected due to it sounding impressive. Choose to by because it fixes the issue you actually have. Tools That Blur the Lines Before you make one decision or the other, it’s worth mentioning some hybrid tools: Dask: Parallelize python across cores or clusters (mini spark).
🔗Dask Documentation


Modin: Multi-threading utilization to speed up Pandas workflows.
🔗Modin Documentation


Polars: Fast DataFrames under the hood with Rust.
🔗Polars Documentation These can be excellent “in-between” solutions when you’ve outgrown basic Python but don’t need full Spark power. Polars is one such lifeboat on medium-sized datasets, and it’s apeshit fast. TL;DR Stick with Python when your data fits in memory and you need speed, simplicity, and flexibility. Switch to Spark when your data is too big, your machine is gasping for air, and your workflows demand distributed processing. Measure your actual needs. Don't scale up just because you can. Scale up because you must. Thanks for reading! Have you ever switched to Spark and regretted it? Or pushed Python way past its limits? I’d love to hear your war stories. Drop your thoughts in the comments, or connect if you’ve got your own scale-up lessons to share. It always starts small. A CSV file here, some JSONs there. You write some good python code using Pandas, maybe some NumPy, and everything runs as expected. Fast. Simple. Beautiful. But then... the files grow. Queries slow down. You get the dreaded “Out of Memory” error. Now all of a sudden, you’re questioning whether to elevate your game. And for many of us, that is one word: Spark. But when is it really worth migrating from the old faithful python to Spark? When does scaling up help solve your problems—and when does it simply complicate everything further? Let’s figure that out. Python: The Single-Machine Workhorse For faster uses, Python has no rivals and allows you to manipulate small to medium datasets. If your entire dataset is comfortably loaded in your machine’s memory (RAM), Python + Pandas or Polars will usually deliver the fastest development time and cleanest code. And let’s be honest: many of us start here because it’s easy. Install Python. Install Pandas. Write a few lines. Done. Why stick with Python? Ease of use. Easy set up, great documentation and big ecosystem. Readable code. Most of the workflows follow intuitively from Pandas, NumPy, and just Python built-ins. Speed for small jobs. Python is fast for datasets that are a couple gigs or smaller. Local development. No clusters, distributed systems, etc. Fast prototyping. Prototype quickly and test concepts without waiting on infrastructure. Ease of use. Easy set up, great documentation and big ecosystem. Ease of use. Easy set up, great documentation and big ecosystem. Ease of use. Readable code. Most of the workflows follow intuitively from Pandas, NumPy, and just Python built-ins. Readable code. Most of the workflows follow intuitively from Pandas, NumPy, and just Python built-ins. Readable code. Speed for small jobs. Python is fast for datasets that are a couple gigs or smaller. Speed for small jobs. Python is fast for datasets that are a couple gigs or smaller. Speed for small jobs. Local development. No clusters, distributed systems, etc. Local development. No clusters, distributed systems, etc. Local development. Fast prototyping. Prototype quickly and test concepts without waiting on infrastructure. Fast prototyping. Prototype quickly and test concepts without waiting on infrastructure. Fast prototyping. 🔗 Pandas Documentation Pandas Documentation 🔗 Polars Documentation Polars Documentation Common Python sweet spots: ETL scripts Data cleaning API data parsing Exploratory Data Analysis (EDA) Prototypes and notebooks Ad hoc data tasks ETL scripts Data cleaning API data parsing Exploratory Data Analysis (EDA) Prototypes and notebooks Ad hoc data tasks But... here's the catch: Python, and therefore Pandas is single-threaded and memory-bound. As soon as you start having your dataset bigger than your RAM, it is getting slow. Really slow. The first indication is usually the sound of your machine’s fan kicking into overdrive as if it were at the beginning of a flight. Then the lag creeps in. You might try chunking the data, or writing a for-loop to process files individually. You begin to sprinkle in “TODO” comments for edge case handling, telling yourself that you’ll “optimize this later.” At some point, Python literally just... gives up. Spark: The Distributed Heavy Lifter Spark was created for big data. We are talking terabytes or petabytes over several machines. The key to Spark’s performance is that it scales horizontally, allowing it to divide work across a cluster of servers and process chunks of your data at the same time. Spark doesn’t care if your laptop has 8GB of RAM, because it solves jobs that require hundreds of gigabytes, or even more. Why move to Spark? Large datasets that cannot be fitted in a single machine. Processing in parallel both across CPUs and across servers. Join multi-levels, group-bys or complex transformations in batch. Built-in fault tolerance. Strong integrations with cloud storage (S3, ADLS, GCS, etc.) Reliable and predictable runs of production-grade batch pipelines. Large datasets that cannot be fitted in a single machine. Large datasets that cannot be fitted in a single machine. Processing in parallel both across CPUs and across servers. Processing in parallel both across CPUs and across servers. Join multi-levels, group-bys or complex transformations in batch. Join multi-levels, group-bys or complex transformations in batch. Built-in fault tolerance. Built-in fault tolerance. Strong integrations with cloud storage (S3, ADLS, GCS, etc.) Strong integrations with cloud storage (S3, ADLS, GCS, etc.) Reliable and predictable runs of production-grade batch pipelines. Reliable and predictable runs of production-grade batch pipelines. 🔗 Apache Spark Documentation Apache Spark Documentation 🔗 PySpark Documentation PySpark Documentation Spark shines when: You are joining several large datasets. You have business-critical jobs that run daily (or hourly). Your data is expanding exponentially and you realize that soon you’ll outgrow a single node. The company already has a cluster, you just need to connect into an existing environment. You are joining several large datasets. You have business-critical jobs that run daily (or hourly). Your data is expanding exponentially and you realize that soon you’ll outgrow a single node. The company already has a cluster, you just need to connect into an existing environment. But (and it’s a big but)… Spark comes with overhead. Setting up Spark and managing clusters, tuning performance, it’s not easy. For small jobs Spark can be slower than Python as the time taken to start the job, distribute the data and shuffle the output can be greater than the benefits provided by Parallelism. 🚩 5 Signs You Need to Scale Your Business Up Not convinced that you should make the switch to Spark? Here are some clear signs your current Python setup may be throwing in the towel: 1️⃣ Out-of-Memory Errors If your scripts are regularly crashing due to memory issues, and you’ve optimized your data types and chunked where possible already … then it might be your time. 2️⃣ Jobs Taking Too Long When a job that once took minutes now takes hours — or worse, does not complete at all — scaling out via distributed processing can help reclaim your time (and sanity). 3️⃣ Frequent Timeouts or Failures Pipelines that arbitrarily error out at resource boundaries are ops hell. These loads are much more graceful with distributed systems like spark. 4️⃣ You’re Splitting Work by Hand If you realize that you are writing loops to manually pick up the data in pieces, process it, and join it back together, that is literally what Spark does on its own—only with more effort and risk. 5️⃣ Companies Growing Rapidly Despite Their Architecture If your company is scaling quickly and data volumes are rising, switching before things break is preferable to switching after. 6️⃣ Your Hardware Scaling Isn’t Making a Difference Upgraded your machine from 16GB to 32GB of RAM, are you still struggling? Single-machine power can only be pushed so far. At times it’s not just about scaling up, but also scaling out. How to Choose: Python VS Spark? Here’s my own cheat sheet: Scenario

Use Python

Use Spark



Dataset < 5GB

✅

❌



Dataset > 10GB

🚩

✅



Quick prototype

✅

❌



Daily production ETL

Maybe

✅



Heavy joins & group-bys

❌

✅



Lots of small API calls

✅

❌



Cluster available

Maybe

✅



Just you on a laptop

✅

❌ Scenario

Use Python

Use Spark Dataset < 5GB

✅

❌ Dataset > 10GB

🚩

✅ Quick prototype

✅

❌ Daily production ETL

Maybe

✅ Heavy joins & group-bys

❌

✅ Lots of small API calls

✅

❌ Cluster available

Maybe

✅ Just you on a laptop

✅

❌ Scenario Use Python Use Spark Scenario Scenario Use Python Use Python Use Spark Use Spark Dataset < 5GB ✅ ❌ Dataset < 5GB Dataset < 5GB ✅ ✅ ❌ ❌ Dataset > 10GB 🚩 ✅ Dataset > 10GB Dataset > 10GB 🚩 🚩 ✅ ✅ Quick prototype ✅ ❌ Quick prototype Quick prototype ✅ ✅ ❌ ❌ Daily production ETL Maybe ✅ Daily production ETL Daily production ETL Maybe Maybe ✅ ✅ Heavy joins & group-bys ❌ ✅ Heavy joins & group-bys Heavy joins & group-bys ❌ ❌ ✅ ✅ Lots of small API calls ✅ ❌ Lots of small API calls Lots of small API calls ✅ ✅ ❌ ❌ Cluster available Maybe ✅ Cluster available Cluster available Maybe Maybe ✅ ✅ Just you on a laptop ✅ ❌ Just you on a laptop Just you on a laptop ✅ ✅ ❌ ❌ When the Wrong Choice Gets Made I've also watched people switch to Spark much too soon. They heard "big data" and believed every CSV required a cluster. But clusters are expensive. Managing them takes time. And when Spark is abused for small datasets, the overhead can destroy performance. On the other end, I do hate seeing failed teams get stuck with complex, memory-heavy Pandas code bases that crash every day… when a small Spark job would’ve done the trick just fine. The moral? The moral? A tool should not be selected due to it sounding impressive. Choose to by because it fixes the issue you actually have. Tools That Blur the Lines Before you make one decision or the other, it’s worth mentioning some hybrid tools: Dask: Parallelize python across cores or clusters (mini spark).
🔗Dask Documentation Modin: Multi-threading utilization to speed up Pandas workflows.
🔗Modin Documentation Polars: Fast DataFrames under the hood with Rust.
🔗Polars Documentation Dask: Parallelize python across cores or clusters (mini spark). 🔗Dask Documentation Dask : Parallelize python across cores or clusters (mini spark). Dask 🔗 Dask Documentation Dask Documentation Modin: Multi-threading utilization to speed up Pandas workflows. 🔗Modin Documentation Modin : Multi-threading utilization to speed up Pandas workflows. Modin 🔗 Modin Documentation Modin Documentation Polars: Fast DataFrames under the hood with Rust. 🔗Polars Documentation Polars : Fast DataFrames under the hood with Rust. Polars 🔗 Polars Documentation Polars Documentation These can be excellent “in-between” solutions when you’ve outgrown basic Python but don’t need full Spark power. Polars is one such lifeboat on medium-sized datasets, and it’s apeshit fast. TL;DR Stick with Python when your data fits in memory and you need speed, simplicity, and flexibility. Switch to Spark when your data is too big, your machine is gasping for air, and your workflows demand distributed processing. Measure your actual needs. Don't scale up just because you can . Scale up because you must . can must Thanks for reading! Have you ever switched to Spark and regretted it? Or pushed Python way past its limits? I’d love to hear your war stories. Drop your thoughts in the comments, or connect if you’ve got your own scale-up lessons to share.

Python vs. Spark: When Does It Make Sense to Scale Up?

Authors:
(1) Jongmin Lee, Department of Mathematical Science, Seoul National University;
(2) Ernest K. Ryu, Department of Mathematical Science, Seoul National University and Interdisciplinary Program in Artificial Intelligence, Seoul National University. Abstract and 1 Introduction 1.1 Notations and preliminaries 1.2 Prior works 2 Anchored Value Iteration 2.1 Accelerated rate for Bellman consistency operator 2.2 Accelerated rate for Bellman optimality opera 3 Convergence when y=1 4 Complexity lower bound 5 Approximate Anchored Value Iteration 6 Gauss–Seidel Anchored Value Iteration 7 Conclusion, Acknowledgments and Disclosure of Funding and References A Preliminaries B Omitted proofs in Section 2 C Omitted proofs in Section 3 D Omitted proofs in Section 4 E Omitted proofs in Section 5 F Omitted proofs in Section 6 G Broader Impacts H Limitations F Omitted proofs in Section 6 Next, we prove following key lemma Proof of Lemma 21. First, we prove first inequality in Lemma 21 by induction. If k= 0, By induction, First, we prove second inequality in Lemma 21 by induction. If k= 0, By induction. Now, we prove the first rate in Theorem 7. For the second rates of Theorem 7, we introduce following lemma. Now, we prove the second rates in Theorem 7. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Jongmin Lee, Department of Mathematical Science, Seoul National University; (2) Ernest K. Ryu, Department of Mathematical Science, Seoul National University and Interdisciplinary Program in Artificial Intelligence, Seoul National University. Authors: Authors: (1) Jongmin Lee, Department of Mathematical Science, Seoul National University; (2) Ernest K. Ryu, Department of Mathematical Science, Seoul National University and Interdisciplinary Program in Artificial Intelligence, Seoul National University. Abstract and 1 Introduction Abstract and 1 Introduction 1.1 Notations and preliminaries 1.1 Notations and preliminaries 1.2 Prior works 1.2 Prior works 2 Anchored Value Iteration 2 Anchored Value Iteration 2.1 Accelerated rate for Bellman consistency operator 2.1 Accelerated rate for Bellman consistency operator 2.2 Accelerated rate for Bellman optimality opera 2.2 Accelerated rate for Bellman optimality opera 3 Convergence when y=1 3 Convergence when y=1 4 Complexity lower bound 4 Complexity lower bound 5 Approximate Anchored Value Iteration 5 Approximate Anchored Value Iteration 6 Gauss–Seidel Anchored Value Iteration 6 Gauss–Seidel Anchored Value Iteration 7 Conclusion, Acknowledgments and Disclosure of Funding and References 7 Conclusion, Acknowledgments and Disclosure of Funding and References A Preliminaries A Preliminaries B Omitted proofs in Section 2 B Omitted proofs in Section 2 C Omitted proofs in Section 3 C Omitted proofs in Section 3 D Omitted proofs in Section 4 D Omitted proofs in Section 4 E Omitted proofs in Section 5 E Omitted proofs in Section 5 F Omitted proofs in Section 6 F Omitted proofs in Section 6 G Broader Impacts G Broader Impacts H Limitations H Limitations F Omitted proofs in Section 6 Next, we prove following key lemma Proof of Lemma 21. First, we prove first inequality in Lemma 21 by induction. If k= 0, By induction, First, we prove second inequality in Lemma 21 by induction. If k= 0, By induction. Now, we prove the first rate in Theorem 7. For the second rates of Theorem 7, we introduce following lemma. Now, we prove the second rates in Theorem 7. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Making Sense of AI Learning Proofs

Bringing Permissionless Lending: Making Sense of Euler

Jan permisompi maytʼasiñ apaniña: Euler chachan amuyupa

El préstamo sin necesidad de autorización: cómo entender el caso Euler

آوردن وام بدون اجازه: ایجاد حس اویلر

Traer préstamos sen permiso: dar sentido a Euler

अनुमति रहित ऋण लाना: यूलर का अर्थ समझना

הבאת הלוואות ללא רשות: הגיוני אוילר

許可なしの融資の実現: オイラーを理解する

Membawa Pinjaman Tanpa Kebenaran: Memahami Euler

අවසරයකින් තොරව ණය දීම: ඉයුලර් පිළිබඳ හැඟීමක් ඇති කිරීම

ፍቓድ ዘይብሉ ልቓሕ ምምጻእ፡ ትርጉም ምሃብ ኦይለር

Rugsat berilmedik karz bermek: Euleriň duýgusyny döretmek

Запровадження бездозвільного кредитування: осмислення Ейлера

Table of Links Abstract and Introduction


Domain and Task
2.1. Data sources and complexity
2.2. Task definition


Related Work
3.1. Text mining and NLP research overview
3.2. Text mining and NLP in industry use
3.3. Text mining and NLP for procurement
3.4. Conclusion from literature review


Proposed Methodology
4.1. Domain knowledge
4.2. Content extraction
4.3. Lot zoning
4.4. Lot item detection
4.5. Lot parsing
4.6. XML parsing, data joining, and risk indices development


Experiment and Demonstration
5.1. Component evaluation
5.2. System demonstration


Discussion
6.1. The ‘industry’ focus of the project
6.2. Data heterogeneity, multilingual and multi-task nature
6.3. The dilemma of algorithmic choices
6.4. The cost of training data


Conclusion, Acknowledgements, and References 4.1. Domain knowledge With the semi-structured data available from TED tender XMLs and award XMLs, Vamstar developed XML parsers that extract different fields and store their unique values in a temporary data store (to be called the ‘tender and award fields’ or just ‘fields’ in short). As an example, given the XML in Figure 1, a record would be created with fields such as: buyer name and addresses, lots (e.g., ‘Verapamil solution for inj. Lot No: 62’ but with multiple values as the notice contains more than 90 lots), award criteria, etc. This allows generation of useful domain knowledge in different ways. First, training data for different fields can be created by simply taking the unique values from each field. These values usually take the form of short phrases or sentences, and therefore, can be used to train text classifiers at phase/sentence level. Note that this data is multilingual. Second, domain lexicons for different fields can be built by extracting ‘representative’ words from each field. Given a field that we are interested in, we merge all values from different languages, and use machine translation tools to translate them into English. Instead of drawing a boundary between what is and is not a ‘representative’ word for that field, here we explain a process to calculate the ‘specificities’ of words and we refer to this as ‘domain specificity lexicon’, which is used in other components of our method. Given a word 𝑤, the basic idea is to measure how unique 𝑤 is to lot/item descriptions compared to other text content in a tender. The basic process is as follows: Identify the set of ‘fields’ that are most relevant to a tender. For this we selected 5 fields - this is rather a subjective interpretation by the domain experts: ‘Name and addresses’ of buyers (Section I.1.1 of the notice in Figure 1), ‘notice title’ (II.1.1), ‘short description’ (II.1.4), ‘lot and item descriptions’ (a concatenation of values from fields like II.2.1), ‘contract criteria’ (a concatenation of values from fields like II.2.5). Following the process mentioned above, all values are then translated into English;


For each field, create a bag-of-words (‘BOW’) representation by concatenating the values of that field across all records in the database, applying stop words removal and lowercasing;


For 𝑤, calculate 4 metrics as follows: 𝑛𝑡𝑓(𝑤) normalised frequency of 𝑤 in the BOW representation of the ‘lot and item descriptions’ field, calculated as the ratio between the frequency of 𝑤 in the BOW over the total words in the BOW; 𝑛𝑑𝑓(𝑤) the normalised ‘document frequency’ of 𝑤, calculated as the ratio between the number of fields in which 𝑤 is found over the number of all fields (i.e., 5); 𝑛𝑡𝑓ᐧ𝑛𝑑𝑓(𝑤) , inspired by the tf-idf measure used in document retrieval, this is the product − of 𝑛𝑡𝑓(𝑤) and the inverse of 𝑛𝑡𝑓(𝑤); 𝑤𝑒𝑖𝑟𝑑𝑛𝑒𝑠𝑠(𝑤), a score calculated by comparing 𝑛𝑡𝑓(𝑤) against that calculated in a reference, general purpose corpus (in this case, the Brown corpus). Thus our domain specificity lexicon contains unique words found within the ‘lot and item descriptions’ field, and each word has four associated scores. In addition to this, we also use another two dictionaries compiled by Vamstar. One is a list of words often used as measurement units (e.g., mg, ml), and the other is a list of words often used to describe the ‘form’ of required items (e.g., pack, box, bottle). Both are very short and contain only a few dozens of entries. These lexicons and dictionaries are also translated to other languages using Google Translate. Authors:
(1) Ziqi Zhang*, Information School, the University of Sheffield, Regent Court, Sheffield, UKS1 4DP (Ziqi.Zhang@sheffield.ac.uk);
(2) Tomas Jasaitis, Vamstar Ltd., London (Tomas.Jasaitis@vamstar.io);
(3) Richard Freeman, Vamstar Ltd., London (Richard.Freeman@vamstar.io);
(4) Rowida Alfrjani, Information School, the University of Sheffield, Regent Court, Sheffield, UKS1 4DP (Rowida.Alfrjani@sheffield.ac.uk);
(5) Adam Funk, Information School, the University of Sheffield, Regent Court, Sheffield, UKS1 4DP (Adam.Funk@sheffield.ac.uk). This paper is available on arxiv under CC BY 4.0 license. Table of Links Abstract and Introduction Domain and Task
2.1. Data sources and complexity
2.2. Task definition Related Work
3.1. Text mining and NLP research overview
3.2. Text mining and NLP in industry use
3.3. Text mining and NLP for procurement
3.4. Conclusion from literature review Proposed Methodology
4.1. Domain knowledge
4.2. Content extraction
4.3. Lot zoning
4.4. Lot item detection
4.5. Lot parsing
4.6. XML parsing, data joining, and risk indices development Experiment and Demonstration
5.1. Component evaluation
5.2. System demonstration Discussion
6.1. The ‘industry’ focus of the project
6.2. Data heterogeneity, multilingual and multi-task nature
6.3. The dilemma of algorithmic choices
6.4. The cost of training data Conclusion, Acknowledgements, and References Abstract and Introduction Abstract and Introduction Abstract and Introduction Abstract and Introduction Domain and Task 2.1. Data sources and complexity 2.2. Task definition Domain and Task Domain and Task 2.1. Data sources and complexity 2.1. Data sources and complexity 2.2. Task definition 2.2. Task definition Related Work 3.1. Text mining and NLP research overview 3.2. Text mining and NLP in industry use 3.3. Text mining and NLP for procurement 3.4. Conclusion from literature review Related Work Related Work 3.1. Text mining and NLP research overview 3.1. Text mining and NLP research overview 3.2. Text mining and NLP in industry use 3.2. Text mining and NLP in industry use 3.3. Text mining and NLP for procurement 3.3. Text mining and NLP for procurement 3.4. Conclusion from literature review 3.4. Conclusion from literature review Proposed Methodology 4.1. Domain knowledge 4.2. Content extraction 4.3. Lot zoning 4.4. Lot item detection 4.5. Lot parsing 4.6. XML parsing, data joining, and risk indices development Proposed Methodology Proposed Methodology Proposed Methodology 4.1. Domain knowledge 4.1. Domain knowledge 4.2. Content extraction 4.2. Content extraction 4.3. Lot zoning 4.3. Lot zoning 4.4. Lot item detection 4.4. Lot item detection 4.5. Lot parsing 4.5. Lot parsing 4.6. XML parsing, data joining, and risk indices development 4.6. XML parsing, data joining, and risk indices development Experiment and Demonstration 5.1. Component evaluation 5.2. System demonstration Experiment and Demonstration Experiment and Demonstration 5.1. Component evaluation 5.1. Component evaluation 5.2. System demonstration 5.2. System demonstration Discussion 6.1. The ‘industry’ focus of the project 6.2. Data heterogeneity, multilingual and multi-task nature 6.3. The dilemma of algorithmic choices 6.4. The cost of training data Discussion Discussion 6.1. The ‘industry’ focus of the project 6.1. The ‘industry’ focus of the project 6.2. Data heterogeneity, multilingual and multi-task nature 6.2. Data heterogeneity, multilingual and multi-task nature 6.3. The dilemma of algorithmic choices 6.3. The dilemma of algorithmic choices 6.4. The cost of training data 6.4. The cost of training data Conclusion, Acknowledgements, and References Conclusion, Acknowledgements, and References Conclusion, Acknowledgements, and References Conclusion, Acknowledgements, and References 4.1. Domain knowledge With the semi-structured data available from TED tender XMLs and award XMLs, Vamstar developed XML parsers that extract different fields and store their unique values in a temporary data store (to be called the ‘tender and award fields’ or just ‘fields’ in short). As an example, given the XML in Figure 1, a record would be created with fields such as: buyer name and addresses, lots (e.g., ‘Verapamil solution for inj. Lot No: 62’ but with multiple values as the notice contains more than 90 lots), award criteria, etc. This allows generation of useful domain knowledge in different ways. First, training data for different fields can be created by simply taking the unique values from each field. These values usually take the form of short phrases or sentences, and therefore, can be used to train text classifiers at phase/sentence level. Note that this data is multilingual. training data Second, domain lexicons for different fields can be built by extracting ‘representative’ words from each field. Given a field that we are interested in, we merge all values from different languages, and use machine translation tools to translate them into English. Instead of drawing a boundary between what is and is not a ‘representative’ word for that field, here we explain a process to calculate the ‘specificities’ of words and we refer to this as ‘domain specificity lexicon’, which is used in other components of our method. Given a word 𝑤, the basic idea is to measure how unique 𝑤 is to lot/item descriptions compared to other text content in a tender. The basic process is as follows: domain lexicons Identify the set of ‘fields’ that are most relevant to a tender. For this we selected 5 fields - this is rather a subjective interpretation by the domain experts: ‘Name and addresses’ of buyers (Section I.1.1 of the notice in Figure 1), ‘notice title’ (II.1.1), ‘short description’ (II.1.4), ‘lot and item descriptions’ (a concatenation of values from fields like II.2.1), ‘contract criteria’ (a concatenation of values from fields like II.2.5). Following the process mentioned above, all values are then translated into English; For each field, create a bag-of-words (‘BOW’) representation by concatenating the values of that field across all records in the database, applying stop words removal and lowercasing; For 𝑤, calculate 4 metrics as follows: 𝑛𝑡𝑓(𝑤) normalised frequency of 𝑤 in the BOW representation of the ‘lot and item descriptions’ field, calculated as the ratio between the frequency of 𝑤 in the BOW over the total words in the BOW; 𝑛𝑑𝑓(𝑤) the normalised ‘document frequency’ of 𝑤, calculated as the ratio between the number of fields in which 𝑤 is found over the number of all fields (i.e., 5); 𝑛𝑡𝑓ᐧ𝑛𝑑𝑓(𝑤) , inspired by the tf-idf measure used in document retrieval, this is the product − of 𝑛𝑡𝑓(𝑤) and the inverse of 𝑛𝑡𝑓(𝑤); 𝑤𝑒𝑖𝑟𝑑𝑛𝑒𝑠𝑠(𝑤), a score calculated by comparing 𝑛𝑡𝑓(𝑤) against that calculated in a reference, general purpose corpus (in this case, the Brown corpus). Identify the set of ‘fields’ that are most relevant to a tender. For this we selected 5 fields - this is rather a subjective interpretation by the domain experts: ‘Name and addresses’ of buyers (Section I.1.1 of the notice in Figure 1), ‘notice title’ (II.1.1), ‘short description’ (II.1.4), ‘lot and item descriptions’ (a concatenation of values from fields like II.2.1), ‘contract criteria’ (a concatenation of values from fields like II.2.5). Following the process mentioned above, all values are then translated into English; Identify the set of ‘fields’ that are most relevant to a tender. For this we selected 5 fields - this is rather a subjective interpretation by the domain experts: ‘Name and addresses’ of buyers (Section I.1.1 of the notice in Figure 1), ‘notice title’ (II.1.1), ‘short description’ (II.1.4), ‘lot and item descriptions’ (a concatenation of values from fields like II.2.1), ‘contract criteria’ (a concatenation of values from fields like II.2.5). Following the process mentioned above, all values are then translated into English; For each field, create a bag-of-words (‘BOW’) representation by concatenating the values of that field across all records in the database, applying stop words removal and lowercasing; For each field, create a bag-of-words (‘BOW’) representation by concatenating the values of that field across all records in the database, applying stop words removal and lowercasing; For 𝑤, calculate 4 metrics as follows: 𝑛𝑡𝑓(𝑤) normalised frequency of 𝑤 in the BOW representation of the ‘lot and item descriptions’ field, calculated as the ratio between the frequency of 𝑤 in the BOW over the total words in the BOW; 𝑛𝑑𝑓(𝑤) the normalised ‘document frequency’ of 𝑤, calculated as the ratio between the number of fields in which 𝑤 is found over the number of all fields (i.e., 5); 𝑛𝑡𝑓ᐧ𝑛𝑑𝑓(𝑤) , inspired by the tf-idf measure used in document retrieval, this is the product − of 𝑛𝑡𝑓(𝑤) and the inverse of 𝑛𝑡𝑓(𝑤); 𝑤𝑒𝑖𝑟𝑑𝑛𝑒𝑠𝑠(𝑤), a score calculated by comparing 𝑛𝑡𝑓(𝑤) against that calculated in a reference, general purpose corpus (in this case, the Brown corpus). For 𝑤, calculate 4 metrics as follows: 𝑛𝑡𝑓(𝑤) normalised frequency of 𝑤 in the BOW representation of the ‘lot and item descriptions’ field, calculated as the ratio between the frequency of 𝑤 in the BOW over the total words in the BOW; 𝑛𝑑𝑓(𝑤) the normalised ‘document frequency’ of 𝑤, calculated as the ratio between the number of fields in which 𝑤 is found over the number of all fields (i.e., 5); 𝑛𝑡𝑓ᐧ𝑛𝑑𝑓(𝑤) , inspired by the tf-idf measure used in document retrieval, this is the product − of 𝑛𝑡𝑓(𝑤) and the inverse of 𝑛𝑡𝑓(𝑤); 𝑤𝑒𝑖𝑟𝑑𝑛𝑒𝑠𝑠(𝑤), a score calculated by comparing 𝑛𝑡𝑓(𝑤) against that calculated in a reference, general purpose corpus (in this case, the Brown corpus). Thus our domain specificity lexicon contains unique words found within the ‘lot and item descriptions’ field, and each word has four associated scores. In addition to this, we also use another two dictionaries compiled by Vamstar. One is a list of words often used as measurement units (e.g., mg, ml), and the other is a list of words often used to describe the ‘form’ of required items (e.g., pack, box, bottle). Both are very short and contain only a few dozens of entries. These lexicons and dictionaries are also translated to other languages using Google Translate. Authors: (1) Ziqi Zhang*, Information School, the University of Sheffield, Regent Court, Sheffield, UKS1 4DP (Ziqi.Zhang@sheffield.ac.uk); (2) Tomas Jasaitis, Vamstar Ltd., London (Tomas.Jasaitis@vamstar.io); (3) Richard Freeman, Vamstar Ltd., London (Richard.Freeman@vamstar.io); (4) Rowida Alfrjani, Information School, the University of Sheffield, Regent Court, Sheffield, UKS1 4DP (Rowida.Alfrjani@sheffield.ac.uk); (5) Adam Funk, Information School, the University of Sheffield, Regent Court, Sheffield, UKS1 4DP (Adam.Funk@sheffield.ac.uk). Authors: Authors: (1) Ziqi Zhang*, Information School, the University of Sheffield, Regent Court, Sheffield, UKS1 4DP (Ziqi.Zhang@sheffield.ac.uk); (2) Tomas Jasaitis, Vamstar Ltd., London (Tomas.Jasaitis@vamstar.io); (3) Richard Freeman, Vamstar Ltd., London (Richard.Freeman@vamstar.io); (4) Rowida Alfrjani, Information School, the University of Sheffield, Regent Court, Sheffield, UKS1 4DP (Rowida.Alfrjani@sheffield.ac.uk); (5) Adam Funk, Information School, the University of Sheffield, Regent Court, Sheffield, UKS1 4DP (Adam.Funk@sheffield.ac.uk). This paper is available on arxiv under CC BY 4.0 license. This paper is available on arxiv under CC BY 4.0 license. available on arxiv available on arxiv

How Vamstar Makes Sense of Complex Tender Data

Hey Hackers!I-Sense has been nominated in HackerNoon's Startups of The Year 2024 awards in Casablanca, Morocco under the AI category. Please vote for us here: https://hackernoon.com/startups/asia/asia-casablanca-morocco?stup=66f316a96ea4677e23d820de&embedable=true Read more about us below to understand why we deserve your vote. Can you tell us about the journey of your startup? What inspired you to go into AI, and what were some of the initial challenges you faced? The journey of I-Sense began with a passion for harnessing AI to solve real-world industrial challenges. We saw how downtime and unpredictable maintenance costs were issues for industries relying on complex machinery. With backgrounds in AI and engineering, we decided to create predictive maintenance solutions that could transform how industries manage their operations, anticipating issues before they become problems. One of our first challenges was building a robust AI model that could adapt across different machines and industries. Establishing trust with early clients was another hurdle, as predictive maintenance was still emerging. At I-Sense, we empower industries with AI-driven predictive maintenance, enabling smarter, cost-efficient, and reliable operations. Our Maintenance 5.0 solutions leverage machine learning to foresee potential equipment failures, reduce downtime, and optimize performance across industrial setups. How has your Startup differentiated itself from other AI solutions in the market? I-Sense stands out in the AI market through its focus on adaptive predictive maintenance. Unlike many one-size-fits-all solutions, I-Sense customizes its AI models to align with the unique machinery and operational needs of each client, maximizing efficiency and reducing downtime in a targeted way. Our platform’s flexibility allows it to integrate across various industrial sectors and machinery types, making it accessible and highly valuable for diverse industries. With a track record of tangible improvements, I-Sense aims to redefine Maintenance 5.0, keeping businesses future-ready and competitive. What have been some of your most significant milestones or achievements to date? Our journey has been marked by a few significant milestones. Early on, we achieved a breakthrough by securing our first large-scale industrial partner, proving our predictive maintenance model's reliability in a real-world setting. This success led to our platform’s rapid adoption across various sectors, with measurable outcomes in downtime reduction and cost savings. A key achievement was reaching over 90% accuracy in predicting equipment failures, a testament to our robust AI model and dedicated team. We’re proud to be setting new standards for AI-driven maintenance! What are your future plans and goals for your Startup? How do you see your AI technology evolving in the next five years? Our vision for I-Sense in the next five years centers on refining our AI to become a fully autonomous predictive maintenance platform. We aim to integrate real-time data insights, enabling even faster and more precise maintenance decisions. Our technology will evolve to incorporate cutting-edge advancements in machine learning and IoT, making it increasingly intuitive and scalable across even more industries. As we continue to grow, we aspire to lead Maintenance 5.0 by developing solutions that support sustainability, reduce waste, and maximize productivity for industries worldwide. Final Thoughts Reflecting on I-Sense’s journey, we’re incredibly proud of how far we’ve come and the industry impact we’ve made with predictive maintenance. Our team’s dedication has allowed us to develop innovative, adaptable solutions that redefine how businesses maintain their critical assets. Moving forward, our focus remains on pushing technological boundaries, expanding to more sectors, and contributing to a sustainable industrial future. Vote for us today! Welcome to HackerNoon’s Startups of The Year 2024 interview series, spotlighting startups from around the world that are changing the way we live, one solution at a time. If you’ve been nominated, create a brand profile and answer these questions here.
Want to nominate a startup? Follow the steps here.Looking for inspiration? Check out how other startups responded to last year’s Startups of The Year interview questions and examples of company responses here, here, and here. Hey Hackers! I-Sense has been nominated in HackerNoon's Startups of The Year 2024 awards in Casablanca, Morocco under the AI category . I-Sense HackerNoon's Startups of The Year 2024 AI category Please vote for us here: Please vote for us here: https://hackernoon.com/startups/asia/asia-casablanca-morocco?stup=66f316a96ea4677e23d820de&embedable=true https://hackernoon.com/startups/asia/asia-casablanca-morocco?stup=66f316a96ea4677e23d820de&embedable=true Read more about us below to understand why we deserve your vote. Can you tell us about the journey of your startup? What inspired you to go into AI, and what were some of the initial challenges you faced? The journey of I-Sense began with a passion for harnessing AI to solve real-world industrial challenges. We saw how downtime and unpredictable maintenance costs were issues for industries relying on complex machinery. With backgrounds in AI and engineering, we decided to create predictive maintenance solutions that could transform how industries manage their operations, anticipating issues before they become problems. One of our first challenges was building a robust AI model that could adapt across different machines and industries. Establishing trust with early clients was another hurdle, as predictive maintenance was still emerging. At I-Sense, we empower industries with AI-driven predictive maintenance, enabling smarter, cost-efficient, and reliable operations. Our Maintenance 5.0 solutions leverage machine learning to foresee potential equipment failures, reduce downtime, and optimize performance across industrial setups. How has your Startup differentiated itself from other AI solutions in the market? I-Sense stands out in the AI market through its focus on adaptive predictive maintenance. Unlike many one-size-fits-all solutions, I-Sense customizes its AI models to align with the unique machinery and operational needs of each client, maximizing efficiency and reducing downtime in a targeted way. Our platform’s flexibility allows it to integrate across various industrial sectors and machinery types, making it accessible and highly valuable for diverse industries. With a track record of tangible improvements, I-Sense aims to redefine Maintenance 5.0, keeping businesses future-ready and competitive. What have been some of your most significant milestones or achievements to date? Our journey has been marked by a few significant milestones. Early on, we achieved a breakthrough by securing our first large-scale industrial partner, proving our predictive maintenance model's reliability in a real-world setting. This success led to our platform’s rapid adoption across various sectors, with measurable outcomes in downtime reduction and cost savings. A key achievement was reaching over 90% accuracy in predicting equipment failures, a testament to our robust AI model and dedicated team. We’re proud to be setting new standards for AI-driven maintenance! What are your future plans and goals for your Startup? How do you see your AI technology evolving in the next five years? Our vision for I-Sense in the next five years centers on refining our AI to become a fully autonomous predictive maintenance platform. We aim to integrate real-time data insights, enabling even faster and more precise maintenance decisions. Our technology will evolve to incorporate cutting-edge advancements in machine learning and IoT, making it increasingly intuitive and scalable across even more industries. As we continue to grow, we aspire to lead Maintenance 5.0 by developing solutions that support sustainability, reduce waste, and maximize productivity for industries worldwide. Final Thoughts Reflecting on I-Sense’s journey, we’re incredibly proud of how far we’ve come and the industry impact we’ve made with predictive maintenance. Our team’s dedication has allowed us to develop innovative, adaptable solutions that redefine how businesses maintain their critical assets. Moving forward, our focus remains on pushing technological boundaries, expanding to more sectors, and contributing to a sustainable industrial future. Vote for us today! Vote for us today ! Vote for us today Welcome to HackerNoon’s Startups of The Year 2024 interview series, spotlighting startups from around the world that are changing the way we live, one solution at a time. If you’ve been nominated, create a brand profile and answer these questions here. Want to nominate a startup? Follow the steps here. Looking for inspiration? Check out how other startups responded to last year’s Startups of The Year interview questions and examples of company responses here, here, and here. Welcome to HackerNoon’s Startups of The Year 2024 interview series, spotlighting startups from around the world that are changing the way we live, one solution at a time. If you’ve been nominated, create a brand profile and answer these questions here . Startups of The Year 2024 create a brand profile create a brand profile create a brand profile here here Want to nominate a startup? Follow the steps here . here here Looking for inspiration? Check out how other startups responded to last year’s Startups of The Year interview questions and examples of company responses here, here, and here. Looking for inspiration? Check out how other startups responded to last year’s Startups of The Year interview questions and examples of company responses here , here , and here . last year’s Startups of The Year interview questions last year’s Startups of The Year interview questions here here here here here here

Meet I-Sense, Startups of the Year 2024 Nominee

HackerNoon: What is your company in 2–5 words? Uros Mikic: Full-service Webflow agency, one subscription. Why is now the time for your company to exist? We’ve built Flow Ninja by leveraging Webflow as a platform to power our product and marketing website and to offer Webflow to our customers. I believe Webflow’s CMS will become the go-to choice for all CMOs. It allows marketing teams to operate independently from engineering departments, providing greater control and flexibility over their websites. Currently, there are many excellent CMS solutions available. However, once your website traffic exceeds 100,000 visits, you need a robust tool to manage your marketing efforts. At this scale, Webflow stands unrivaled. Moreover, the ongoing challenge of expanding marketing teams involves hiring roles that are not required full-time. Our company addresses this issue by offering an entire team of design, development, and growth experts for the price of a single employee, all included in one subscription. What do you love about your team, and why are you the ones to solve this problem? I’m truly grateful that the team has been with me since our company was founded and that trust is still going strong. As I was 19 when we started, there were many bumps in the road, and were it not for the team surrounding me, it would have been a much more difficult journey. We were among the first to start working in Webflow in 2015, around two years after it was launched. If you weren’t building your startup, what would you be doing? I’ve fallen in love with the game of growing businesses, and I think this will stick with me forever. It gives me a sense of purpose. It makes me seize the day. If it wasn’t for Flow Ninja, I genuinely believe I would look for a potential partnership to help another business or start a new one from scratch. I must admit it: I’ve been daydreaming about running a traditional business. I think everything we do in the startup world would be ten times more effective if applied to a traditional mom-and-pop shop type of business. Introducing all of the systems related to HR, sales, marketing etc, would allow me to grow a traditional business to new heights. One thing I would love is to spend more time in my garden. Suppose time were to stand still, relieving me of all accountability in my business. In that case, I'd likely indulge in ample moments tending to my garden, allowing my mind to unwind and rejuvenate At the moment, how do you measure success? What are your metrics? We have different metrics for different departments. We try to hold people accountable for specific metrics. COO & CPO Customer churn
Team attrition
Company billability CTO & Design Late tickets
Billability
Client initiatives
Client reviews Finance & Sales Number of closed leads
Number of daily reachouts to cold clients
Cashflow (We are a fan of profit-first)
Net profit margin
Projected vs actualized revenue In a few sentences, what do you offer to whom? We offer a full-service Webflow team, including strategy, design, development, QA, SEO, account management, and more — all within a single subscription via our custom client relationship management platform. Our clients are usually product-led growth companies with more than $20M in revenue or funding and marketing teams of more than five people. We also work with enterprises and sales-led companies. What’s most exciting about your traction to date? As I write this, we are entering the most exciting period. As a profitable business, we are currently heavily investing in our team and strategic roles. Still, one of my favorite accolades to date is the 2023 Webflow Enterprise Partner of the Year, awarded by Webflow. Where do you think your growth will be next year? We hope to increase our growth from 30% to 60% by the end of this year and expand even more next year. We recently onboarded a new sales team and a Chief Revenue Officer, so I am confident we could grow much more than this. Tell us about your first paying customer and revenue expectations over the next year. Ah, the first paying customer. No one likes to hear about that. It was a website for an education institution that was built using a simple template. Another agency outsourced this project to us, presenting our work as theirs. They wanted the template to look like they'd built everything custom. On top of that, their budget barely covered our basic expenses, so we worked for peanuts, basically. There were countless revisions, and everything took five times longer than expected. No one quit in the process, and everyone was happy in the end. Those were rookie mistakes, but you live and learn, I guess. As for the expectations — we have aggressive plans to scale to $15M+ in the next five years. We’ve set a cornerstone for that: we focus on our custom-built in-house client relations platform and concentrate on our product. Finally, we ensure that our delivery puts us among the top 1% of all agencies offering similar services. With all that working well, we can now safely invest in growth. We expect to be in the $2-3M range next year. What’s your biggest threat? There are a lot of threats, and we keep them at bay. One might think that the biggest threat to the business is that we rely solely on Webflow to develop all of our websites. But I strongly disagree. I truly believe that Webflow is the only competitive platform in the complex website builders market. I’m confident I can push all of our developments into that platform, and I sleep well because I’ve met every single member of the Webflow leadership team, and they really know what they are doing. HackerNoon : What is your company in 2–5 words? HackerNoon Uros Mikic : Full-service Webflow agency, one subscription. Uros Mikic Uros Mikic Why is now the time for your company to exist? We’ve built Flow Ninja by leveraging Webflow as a platform to power our product and marketing website and to offer Webflow to our customers. I believe Webflow’s CMS will become the go-to choice for all CMOs. It allows marketing teams to operate independently from engineering departments, providing greater control and flexibility over their websites. Flow Ninja Currently, there are many excellent CMS solutions available. However, once your website traffic exceeds 100,000 visits, you need a robust tool to manage your marketing efforts. At this scale, Webflow stands unrivaled. Moreover, the ongoing challenge of expanding marketing teams involves hiring roles that are not required full-time. Our company addresses this issue by offering an entire team of design, development, and growth experts for the price of a single employee, all included in one subscription. What do you love about your team, and why are you the ones to solve this problem? I’m truly grateful that the team has been with me since our company was founded and that trust is still going strong. As I was 19 when we started, there were many bumps in the road, and were it not for the team surrounding me, it would have been a much more difficult journey. We were among the first to start working in Webflow in 2015, around two years after it was launched. If you weren’t building your startup, what would you be doing? I’ve fallen in love with the game of growing businesses, and I think this will stick with me forever. It gives me a sense of purpose. It makes me seize the day. If it wasn’t for Flow Ninja, I genuinely believe I would look for a potential partnership to help another business or start a new one from scratch. I must admit it: I’ve been daydreaming about running a traditional business. I think everything we do in the startup world would be ten times more effective if applied to a traditional mom-and-pop shop type of business. Introducing all of the systems related to HR, sales, marketing etc, would allow me to grow a traditional business to new heights. One thing I would love is to spend more time in my garden. Suppose time were to stand still, relieving me of all accountability in my business. In that case, I'd likely indulge in ample moments tending to my garden, allowing my mind to unwind and rejuvenate At the moment, how do you measure success? What are your metrics? We have different metrics for different departments. We try to hold people accountable for specific metrics. COO & CPO COO & CPO Customer churn Team attrition Company billability Customer churn Team attrition Company billability CTO & Design CTO & Design Late tickets Billability Client initiatives Client reviews Late tickets Billability Client initiatives Client reviews Finance & Sales Finance & Sales Number of closed leads Number of daily reachouts to cold clients Cashflow (We are a fan of profit-first) Net profit margin Projected vs actualized revenue Number of closed leads Number of daily reachouts to cold clients Cashflow (We are a fan of profit-first) Net profit margin Projected vs actualized revenue In a few sentences, what do you offer to whom? We offer a full-service Webflow team, including strategy, design, development, QA, SEO, account management, and more — all within a single subscription via our custom client relationship management platform. Our clients are usually product-led growth companies with more than $20M in revenue or funding and marketing teams of more than five people. We also work with enterprises and sales-led companies. What’s most exciting about your traction to date? As I write this, we are entering the most exciting period. As a profitable business, we are currently heavily investing in our team and strategic roles. Still, one of my favorite accolades to date is the 2023 Webflow Enterprise Partner of the Year, awarded by Webflow. Where do you think your growth will be next year? We hope to increase our growth from 30% to 60% by the end of this year and expand even more next year. We recently onboarded a new sales team and a Chief Revenue Officer, so I am confident we could grow much more than this. Tell us about your first paying customer and revenue expectations over the next year. Ah, the first paying customer. No one likes to hear about that. It was a website for an education institution that was built using a simple template. Another agency outsourced this project to us, presenting our work as theirs. They wanted the template to look like they'd built everything custom. On top of that, their budget barely covered our basic expenses, so we worked for peanuts, basically. There were countless revisions, and everything took five times longer than expected. No one quit in the process, and everyone was happy in the end. Those were rookie mistakes, but you live and learn, I guess. As for the expectations — we have aggressive plans to scale to $15M+ in the next five years. We’ve set a cornerstone for that: we focus on our custom-built in-house client relations platform and concentrate on our product. Finally, we ensure that our delivery puts us among the top 1% of all agencies offering similar services. With all that working well, we can now safely invest in growth. We expect to be in the $2-3M range next year. What’s your biggest threat? There are a lot of threats, and we keep them at bay. One might think that the biggest threat to the business is that we rely solely on Webflow to develop all of our websites. But I strongly disagree. I truly believe that Webflow is the only competitive platform in the complex website builders market. I’m confident I can push all of our developments into that platform, and I sleep well because I’ve met every single member of the Webflow leadership team, and they really know what they are doing.

"Growing Businesses Gives Me a Sense of Purpose." says Flow Ninja Founder/CEO 

"Οι αναπτυσσόμενες επιχειρήσεις μου δίνει μια αίσθηση σκοπού." λέει ο Flow Ninja Founder/CEO 

"Hacer crecer mi negocio me da un sentido de propósito", afirma el fundador y director ejecutivo de Flow Ninja

«Աճող բիզնեսները ինձ նպատակի զգացում են տալիս»: ասում է Flow Ninja-ի հիմնադիր/գործադիր տնօրենը 

"גידול עסקים נותן לי תחושת מטרה." אומר מייסד/מנכ"ל Flow Ninja 

「ビジネスの成長は私に目的意識を与えてくれます」とFlow Ninjaの創設者兼CEOは語る

"ການເຕີບໃຫຍ່ຂອງທຸລະກິດເຮັດໃຫ້ຂ້ອຍມີຄວາມຕັ້ງໃຈ." Flow Ninja ຜູ້ກໍ່ຕັ້ງ / CEO 

„Rozwój firm daje mi poczucie celu” – mówi założyciel/dyrektor generalny Flow Ninja

«Развитие бизнеса дает мне чувство цели», — говорит основатель и генеральный директор Flow Ninja

"Gukura ubucuruzi bimpa kumva intego." ati Flow Ninja washinze / Umuyobozi mukuru 

"வளரும் வணிகங்கள் எனக்கு ஒரு நோக்கத்தை அளிக்கிறது." Flow Ninja நிறுவனர்/CEO கூறுகிறார் 

"ምዕባለ ንግዳዊ ትካላት ናይ ዕላማ ስምዒት ይህበኒ።" ይብል መስራቲ/ዋና ኣካያዲ ስራሕ ፍሎው ኒንጃ 

"İşletmeleri Büyütmek Bana Bir Amaç Duygusu Veriyor." diyor Flow Ninja Kurucusu/CEO'su

SINGAPORE, SG, September 10th, 2024/CyberNewsWire/--Groundbreaking technology integrates Post-Quantum Cryptography and SSL security for the next generation of Face-based eID solutions and Face-protected Digital Public Key Infrastructure (DPKI) Seventh Sense, a pioneer in advanced cybersecurity solutions, announces the launch of SenseCrypt, a revolutionary new platform that sets a new standard in secure, privacy-preserving identity verification. SenseCrypt introduces a first-of-its-kind face-based public key infrastructure (PKI) and electronic identity (eID) solution. This cutting-edge innovation combines Post-Quantum Cryptography (PQC)* — designed to withstand the security threats posed by future quantum computing—with the trusted SSL technology that secures websites worldwide. As the demand for secure identity verification systems rises, SenseCrypt offers a comprehensive solution that tackles both privacy and security challenges. Transforming QR Codes, Face Verifiable Credentials, and Face-protected Encryption At the core of SenseCrypt is a fundamental shift in the identity paradigm to the realm of cryptography. Instead of traditional methods that rely on storing and matching biometrics, SenseCrypt eID utilizes acts of encryption and decryption for registration and authentication, with no public/private keys stored anywhere. This patented approach generates eIDs as encrypted raw bytes, known as SensePrints, which can be printed as QR Codes on various identification mediums including ID cards, documents, and birth certificates, or stored in NFC chips or databases. This unique feature allows for offline verification capability, making it adaptable to various environments. Unlike other solutions available in the market, the QR codes generated do not contain any biometric data. This means that even in the event of compromised keys, no biometrics or personally identifiable information (PII) is at risk, significantly reducing the regulatory and compliance burdens associated with data breaches. Unmatched Verification Accuracy SenseCrypt has a verification accuracy that is unmatched in the industry, with a False Accept Rate (FAR) of 0 with a False Rejection Rate (FRR) of below one percent (<1%)**. For governments, ID providers, and security-focused organizations, even a FAR as low as one-in-a-million (1e-06) is unacceptable when performing millions of verifications daily. The foundational features of SensePrints include: Privacy-preserving and Non-biometric: No biometric data is stored Biometrically verifiable: Only the eID holder’s live face can decrypt the eID attributes Revocable and renewable: Multiple SensePrints can be generated from the same image and with the same metadata. While users cannot change their face, they can change their SensePrint Offline verifiable: SensePrints can be verified completely offline Incomparable and Unlinkable: No biometric data is stored, making it impossible to compare and link if two SensePrints belong to the same individual Group verification capable: A single SensePrint can verify multiple users Irreversible: The absence of biometrics prevents Hill Climbing or face regeneration attacks Non-repudiable: Each SensePrint is signed by the issuer, allowing verification via the issuer’s root certificate public key Advancing Security with Face PKI While SensePrint eID requires the holder’s live face to be present at every transaction, Face PKI allows verifiers to conduct transactions without ever seeing the holder’s face or processing any biometrics. This is achieved through the generation of Face Certificates for specific purposes, such as login, eKYC, and more. These certificates are standard X.509v3 certificates that use a face-derived public key, signed by a trusted issuer. A user can have any number of Face Certificates, each with a unique public key for specific purposes. Face PKI not only enhances traditional security measures but also unlocks new potential use cases such as file and document signing, encryption/decryption, eKYC, passwordless login, proof of presence, secure communication, and multi-factor authentication (MFA). Decentralized Identity with Distributed Ledger Technology Protocol (DLT Protocol) SenseCrypt Face PKI supports various scenarios but relies on a central root of trust. For those preferring a decentralized model and identity wallets (Self-Sovereign Identity), the SenseCrypt DLT Protocol brings all the capabilities of Face PKI to the blockchain. This innovative protocol enables issuers to verify wallet holders without transferring or storing any biometrics on the blockchain or inside wallets, ensuring user privacy through Zero Knowledge Face Proofs. Similarly, verifiers receive cryptographic proof that the wallet's legitimate owner is making the Verifiable Presentation, not someone using a stolen wallet. With the NIST’s standardization of PQC published in August 2024, SenseCrypt Face PKI supports both FIPS 203 – ML-KEM Standard and FIPS 204 – ML-DSA Standard. https://www.whitehouse.gov/oncd/briefing-room/2024/08/13/fact-sheet-biden-harris-administration-continues-work-to-secure-a-post-quantum-cryptography-future/ ** Measured on a proprietary identity dataset of >50 million About Seventh Sense Seventh Sense is a Singapore-based deep-tech company, tackling complex challenges at the intersection of machine learning and cryptography. Its mission is to deliver next-generation identity technology to governments and organizations, supporting both centralized and decentralized systems. The company's stakeholders include government organizations and bodies from both Singapore and Australia. For more information, readers can please contact: Product Licenses: sales@seventhsense.ai
Careers: careers@seventhsense.ai
Investor Relations: ir@seventhsense.ai Contact Communications Manager Shruti Singh Seventh Sense AI shruti@seventhsense.ai This story was distributed as a release by Cyberwire under HackerNoon’s Business Blogging Program. Learn more about the program here. SINGAPORE, SG, September 10th, 2024/CyberNewsWire/--Groundbreaking technology integrates Post-Quantum Cryptography and SSL security for the next generation of Face-based eID solutions and Face-protected Digital Public Key Infrastructure (DPKI) Seventh Sense, a pioneer in advanced cybersecurity solutions, announces the launch of SenseCrypt, a revolutionary new platform that sets a new standard in secure, privacy-preserving identity verification. SenseCrypt introduces a first-of-its-kind face-based public key infrastructure (PKI) and electronic identity (eID) solution. This cutting-edge innovation combines Post-Quantum Cryptography (PQC)* — designed to withstand the security threats posed by future quantum computing—with the trusted SSL technology that secures websites worldwide. As the demand for secure identity verification systems rises, SenseCrypt offers a comprehensive solution that tackles both privacy and security challenges. Transforming QR Codes, Face Verifiable Credentials, and Face-protected Encryption At the core of SenseCrypt is a fundamental shift in the identity paradigm to the realm of cryptography. Instead of traditional methods that rely on storing and matching biometrics, SenseCrypt eID utilizes acts of encryption and decryption for registration and authentication, with no public/private keys stored anywhere. This patented approach generates eIDs as encrypted raw bytes, known as SensePrints, which can be printed as QR Codes on various identification mediums including ID cards, documents, and birth certificates, or stored in NFC chips or databases. This unique feature allows for offline verification capability, making it adaptable to various environments. Unlike other solutions available in the market, the QR codes generated do not contain any biometric data. This means that even in the event of compromised keys, no biometrics or personally identifiable information (PII) is at risk, significantly reducing the regulatory and compliance burdens associated with data breaches. Unmatched Verification Accuracy SenseCrypt has a verification accuracy that is unmatched in the industry, with a False Accept Rate (FAR) of 0 with a False Rejection Rate (FRR) of below one percent (<1%)**. For governments, ID providers, and security-focused organizations, even a FAR as low as one-in-a-million (1e-06) is unacceptable when performing millions of verifications daily. The foundational features of SensePrints include: Privacy-preserving and Non-biometric: No biometric data is stored Biometrically verifiable: Only the eID holder’s live face can decrypt the eID attributes Revocable and renewable: Multiple SensePrints can be generated from the same image and with the same metadata. While users cannot change their face, they can change their SensePrint Offline verifiable: SensePrints can be verified completely offline Incomparable and Unlinkable: No biometric data is stored, making it impossible to compare and link if two SensePrints belong to the same individual Group verification capable: A single SensePrint can verify multiple users Irreversible: The absence of biometrics prevents Hill Climbing or face regeneration attacks Non-repudiable: Each SensePrint is signed by the issuer, allowing verification via the issuer’s root certificate public key Advancing Security with Face PKI While SensePrint eID requires the holder’s live face to be present at every transaction, Face PKI allows verifiers to conduct transactions without ever seeing the holder’s face or processing any biometrics. This is achieved through the generation of Face Certificates for specific purposes, such as login, eKYC, and more. These certificates are standard X.509v3 certificates that use a face-derived public key, signed by a trusted issuer. A user can have any number of Face Certificates, each with a unique public key for specific purposes. Face PKI not only enhances traditional security measures but also unlocks new potential use cases such as file and document signing, encryption/decryption, eKYC, passwordless login, proof of presence, secure communication, and multi-factor authentication (MFA). Decentralized Identity with Distributed Ledger Technology Protocol (DLT Protocol) SenseCrypt Face PKI supports various scenarios but relies on a central root of trust. For those preferring a decentralized model and identity wallets (Self-Sovereign Identity), the SenseCrypt DLT Protocol brings all the capabilities of Face PKI to the blockchain. This innovative protocol enables issuers to verify wallet holders without transferring or storing any biometrics on the blockchain or inside wallets, ensuring user privacy through Zero Knowledge Face Proofs. Similarly, verifiers receive cryptographic proof that the wallet's legitimate owner is making the Verifiable Presentation, not someone using a stolen wallet. With the NIST’s standardization of PQC published in August 2024, SenseCrypt Face PKI supports both FIPS 203 – ML-KEM Standard and FIPS 204 – ML-DSA Standard. https://www.whitehouse.gov/oncd/briefing-room/2024/08/13/fact-sheet-biden-harris-administration-continues-work-to-secure-a-post-quantum-cryptography-future/ ** Measured on a proprietary identity dataset of >50 million https://www.whitehouse.gov/oncd/briefing-room/2024/08/13/fact-sheet-biden-harris-administration-continues-work-to-secure-a-post-quantum-cryptography-future/ About Seventh Sense Seventh Sense is a Singapore-based deep-tech company, tackling complex challenges at the intersection of machine learning and cryptography. Its mission is to deliver next-generation identity technology to governments and organizations, supporting both centralized and decentralized systems. The company's stakeholders include government organizations and bodies from both Singapore and Australia. For more information, readers can please contact: Product Licenses: sales@seventhsense.ai Careers: careers@seventhsense.ai Investor Relations: ir@seventhsense.ai sales@seventhsense.ai careers@seventhsense.ai ir@seventhsense.ai Contact Communications Manager Shruti Singh Seventh Sense AI shruti@seventhsense.ai shruti@seventhsense.ai This story was distributed as a release by Cyberwire under HackerNoon’s Business Blogging Program. Learn more about the program here. This story was distributed as a release by Cyberwire under HackerNoon’s Business Blogging Program. Learn more about the program here . here here

Seventh Sense Unveils Privacy-Preserving Face-Based Public Key Infrastructure And eID Solution

Seventh Sense presenta una infraestructura de clave pública basada en reconocimiento facial y una solución de identificación electrónica que preserva la privacidad

Seventh Sense-k pribatutasuna zaintzeko aurpegian oinarritutako gako publikoen azpiegitura eta eID irtenbidea aurkezten ditu

Seventh Sense dévoile une infrastructure à clé publique basée sur le visage et une solution d'identification électronique préservant la confidentialité

सेवेंथ सेंस ने गोपनीयता-संरक्षण हेतु फेस-आधारित सार्वजनिक कुंजी अवसंरचना और ईआईडी समाधान का अनावरण किया

החוש השביעי חושף תשתית מפתח ציבורי מבוסס-פנים לשמירה על פרטיות ופתרון eID

セブンスセンス、プライバシー保護機能を備えた顔認証ベースの公開鍵基盤とeIDソリューションを発表

მეშვიდე გრძნობა ავლენს კონფიდენციალურობის დაცვას სახის საჯარო გასაღების ინფრასტრუქტურას და eID გადაწყვეტას

Seventh Sense құпиялылықты сақтайтын бетке негізделген ашық кілт инфрақұрылымын және eID шешімін ашады

Seventh Sense onthult privacybeschermende gezichtsgebaseerde openbare sleutelinfrastructuur en eID-oplossing

Седмо чуло открива инфраструктуру јавног кључа засновану на лицу и решење за еИД који чува приватност

Ettinchi Sense Maxfiylikni saqlaydigan yuzga asoslangan ochiq kalitlar infratuzilmasi va eID yechimini taqdim etadi

Seventh Sense 推出基于面部识别技术的隐私保护公钥基础设施和 eID 解决方案

At my work I regularly interviewed managers of various levels – from interns to CPO-level people, and conducted mock interviews. Recently, I went through interviews by myself at Google and other companies and received an offer from one big company in the Middle East. In this article, I will share my experience and give some tips how to prepare for a product interview. The Product Sense Interview or Product Design Interview is an important stage in the interview process for a product manager role. It tests your work experience, the depth of your practical knowledge, and how clearly you can communicate what you know. You need to prepare for all sections, but I would recommend to pay special attention to this one, since this is a key stage of evaluating your product skills. Types of cases Create a product from scratch Example questions – Design a refrigerator for children, Design a taxi service for elder people, Design a vending machine for hotels. These questions may seem too academic, but they are really asked in interviews. For example, I was asked the last question in an interview at Google. Improve an existing product Example questions – Imagine that you are a product manager at Google Search and your task is to improve it, how will you act? This is a case that I always give in my interviews, substituting the product that I am currently working on. Both types of questions can be prepared for using a similar approach. How to prepare Step 1. What to read There are two good books for preparing for interviews that have already become industry standards, and they really help prepare well, not only for product interviews, but also for other types of questions that you will be asked. Cracking PM Interview
I used this book to prepare for interviews at Yandex six years ago. And it's still relevant, I recommend it to all my mentees. It's suitable for any level – both for those who don't know where to start learning the profession of a product manager, and for those who have already advanced in the profession. This book is good because it provides an overview of all the domains that managers need to know, and therefore all the types of questions that need to be prepared. It provides solutions with a clear structure.


Decode & Conquer
It takes second place in my rating and rather complements the first book. It also provides cases with solutions, but it largely duplicates Cracking PM Interview and provides a more complex structure for solving cases, but it's still useful as an additional resource. Step 2. What to watch Watch how other managers go through interviews. You can find many examples on YouTube by searching for Product Design Interview Mockup. For example, here and here. Step 3. Prepare a template for your answer After reading books and watching different interviews, create your own template for your answer, write it down on a piece of paper and keep it handy during the interview, like a cheat sheet. This will help you not to get lost in your thoughts and remember the aspects that need to be covered in your answer. Here is the template I use: Goals (why)
Metrics
Constraints (deadline, resources, platforms, geo)
Users Groups (who, why do they need it, choose one group)
CJM & pain points
Prioritise (reach, impact, confident, effort)
List solutions
Evaluate solutions (trade-off & risks)
Describe your product (sum-up) Step 4. Take a few interviews in advance To avoid being too nervous during real interviews, practice them in advance. The easiest way is to take one of the questions and record a zoom call with yourself, then watch it and analyze the mistakes. More effective is to find a mentor and work through the product section with him, so you will get feedback faster. I recommend taking a few interviews with different people and see what they pay attention to. Hiring managers are very different, it is useful to listen to several opinions in order to better prepare. Tips, what you can’t find in books Before drafting a solution, answer the question “Why?” I always try to think about any product from two sides – why does the business need it and why do the users need it. This also helps in solving cases. The answers to these questions influence the solutions that should be offered. For example, a business may have a main goal of increasing revenue, or it may be growth and retention of the audience. Depending on the goal, the solutions will be different. Make your answer easy to follow A pre-prepared answer plan will help you with this. Some candidates share the screen and sketch out the structure of their answer in real-time. This is great for helping the interviewer follow your thoughts. But this is not always convenient, for example, I type slowly in English, and writing the structure takes time, of which there is not much time during an interview. Therefore, I use another approach. During my response, I try not only to answer the question, but also to tell the plan of my answer upfront, and summarize it at the end. This rule is used in presentations: 1) tell what you are going to say, 2) say it, 3) summarize what you just talked about. Conduct a dialogue A case is not a monologue, it is also designed to demonstrate your communication, cooperation and teamwork skills. Ask clarifying questions, periodically check whether you are on the right track in your solution. But do not try to force the interviewer to solve the case for you. It is normal to ask whether what you are proposing makes sense and whether it is possible to move on to the next stage, but you should not ask what possible solutions exist. Distribute your time properly Typically, an interview lasts 45 minutes, of which five are spent on greeting and a short introduction, and another five at the end on questions about the position you are interviewing for. There are 35 minutes left to solve the case. It’s important to distribute this time properly. From several user groups, it is better to choose one and focus on it. Then identify several problems that this group faces and choose one again. But it is better to offer several solutions. This way you will increase the probability that one of them will be good and will be able to demonstrate prioritization skills. When choosing one of the user groups and one problem, it is better to explain why you decided to focus on it. For example, it could be the largest group or the one that brings in the most revenue. Common mistakes Let me share some common mistakes that I've encountered in my practice. Jump straight to solutions The most common mistake is to jump straight to sketching out ideas. A candidate is asked to improve a product, they open a website or application and starts listing ideas – add a filter here, and a button there. This is a 100% no hire, because it is unlikely that a candidate will be able to come up with something brilliant right away, without knowledge of the product, audience and business goals, rather he will demonstrate a lack of experience. Think that deep expertise in the product is necessary In fact, it doesn't matter what kind of product you are creating or improving – a delivery application, a vending machine or Google search. In solving a case, it is important to show that you know the product approach and no matter what service you are working on, you know where to start. Follow templates too literally Once, I failed an interview because I followed a prepared template too literally instead of solving a case from my head. Solving a case requires not only formally going through all the steps from the template, but also the ability to rely on your own experience and adapt to changing task conditions on the fly. Therefore, it makes sense to solve several cases in advance to practice all the skills together. Conclusion It’s important to be prepared for all interviews, but I would pay special attention to the Product Sense Interview, as this is a key stage, where a product manager can demonstrate their skills. To be well-prepared, I would recommend reading Cracking PM Interview, watching videos of others going through the interview on YouTube and taking few mock interviews in advance. Also it’s important to prepare your own template for an answer and know some common mistakes that candidates often make. I hope this article helps you in your preparation, and wish you success in your interviews. I’m also curious about the challenges you’ve faced during the interview process, and what things helped you went through the product case interview. At my work I regularly interviewed managers of various levels – from interns to CPO-level people, and conducted mock interviews. Recently, I went through interviews by myself at Google and other companies and received an offer from one big company in the Middle East. In this article, I will share my experience and give some tips how to prepare for a product interview. The Product Sense Interview or Product Design Interview is an important stage in the interview process for a product manager role. It tests your work experience, the depth of your practical knowledge, and how clearly you can communicate what you know. You need to prepare for all sections, but I would recommend to pay special attention to this one, since this is a key stage of evaluating your product skills. Types of cases Create a product from scratch Create a product from scratch Example questions – Design a refrigerator for children, Design a taxi service for elder people, Design a vending machine for hotels. These questions may seem too academic, but they are really asked in interviews. For example, I was asked the last question in an interview at Google. Improve an existing product Improve an existing product Example questions – Imagine that you are a product manager at Google Search and your task is to improve it, how will you act? This is a case that I always give in my interviews, substituting the product that I am currently working on. Both types of questions can be prepared for using a similar approach. How to prepare Step 1. What to read Step 1. What to read There are two good books for preparing for interviews that have already become industry standards, and they really help prepare well, not only for product interviews, but also for other types of questions that you will be asked. Cracking PM Interview
I used this book to prepare for interviews at Yandex six years ago. And it's still relevant, I recommend it to all my mentees. It's suitable for any level – both for those who don't know where to start learning the profession of a product manager, and for those who have already advanced in the profession. This book is good because it provides an overview of all the domains that managers need to know, and therefore all the types of questions that need to be prepared. It provides solutions with a clear structure. Decode & Conquer
It takes second place in my rating and rather complements the first book. It also provides cases with solutions, but it largely duplicates Cracking PM Interview and provides a more complex structure for solving cases, but it's still useful as an additional resource. Cracking PM Interview I used this book to prepare for interviews at Yandex six years ago. And it's still relevant, I recommend it to all my mentees. It's suitable for any level – both for those who don't know where to start learning the profession of a product manager, and for those who have already advanced in the profession. This book is good because it provides an overview of all the domains that managers need to know, and therefore all the types of questions that need to be prepared. It provides solutions with a clear structure. Cracking PM Interview Cracking PM Interview I used this book to prepare for interviews at Yandex six years ago. And it's still relevant, I recommend it to all my mentees. It's suitable for any level – both for those who don't know where to start learning the profession of a product manager, and for those who have already advanced in the profession. This book is good because it provides an overview of all the domains that managers need to know, and therefore all the types of questions that need to be prepared. It provides solutions with a clear structure. Decode & Conquer It takes second place in my rating and rather complements the first book. It also provides cases with solutions, but it largely duplicates Cracking PM Interview and provides a more complex structure for solving cases, but it's still useful as an additional resource. Decode & Conquer Decode & Conquer It takes second place in my rating and rather complements the first book. It also provides cases with solutions, but it largely duplicates Cracking PM Interview and provides a more complex structure for solving cases, but it's still useful as an additional resource. Step 2. What to watch Step 2. What to watch Watch how other managers go through interviews. You can find many examples on YouTube by searching for Product Design Interview Mockup. For example, here and here . here here Step 3. Prepare a template for your answer After reading books and watching different interviews, create your own template for your answer, write it down on a piece of paper and keep it handy during the interview, like a cheat sheet. This will help you not to get lost in your thoughts and remember the aspects that need to be covered in your answer. Here is the template I use: Here is the template I use: Goals (why) Metrics Constraints (deadline, resources, platforms, geo) Users Groups (who, why do they need it, choose one group) CJM & pain points Prioritise (reach, impact, confident, effort) List solutions Evaluate solutions (trade-off & risks) Describe your product (sum-up) Goals (why) Metrics Constraints (deadline, resources, platforms, geo) Users Groups (who, why do they need it, choose one group) CJM & pain points Prioritise (reach, impact, confident, effort) List solutions Evaluate solutions (trade-off & risks) Describe your product (sum-up) Step 4. Take a few interviews in advance To avoid being too nervous during real interviews, practice them in advance. The easiest way is to take one of the questions and record a zoom call with yourself, then watch it and analyze the mistakes. More effective is to find a mentor and work through the product section with him, so you will get feedback faster. I recommend taking a few interviews with different people and see what they pay attention to. Hiring managers are very different, it is useful to listen to several opinions in order to better prepare. Tips, what you can’t find in books Before drafting a solution, answer the question “Why?” Before drafting a solution, answer the question “Why?” I always try to think about any product from two sides – why does the business need it and why do the users need it. This also helps in solving cases. The answers to these questions influence the solutions that should be offered. For example, a business may have a main goal of increasing revenue, or it may be growth and retention of the audience. Depending on the goal, the solutions will be different. Make your answer easy to follow Make your answer easy to follow A pre-prepared answer plan will help you with this. Some candidates share the screen and sketch out the structure of their answer in real-time. This is great for helping the interviewer follow your thoughts. But this is not always convenient, for example, I type slowly in English, and writing the structure takes time, of which there is not much time during an interview. Therefore, I use another approach. During my response, I try not only to answer the question, but also to tell the plan of my answer upfront, and summarize it at the end. This rule is used in presentations: 1) tell what you are going to say, 2) say it, 3) summarize what you just talked about. Conduct a dialogue Conduct a dialogue A case is not a monologue, it is also designed to demonstrate your communication, cooperation and teamwork skills. Ask clarifying questions, periodically check whether you are on the right track in your solution. But do not try to force the interviewer to solve the case for you. It is normal to ask whether what you are proposing makes sense and whether it is possible to move on to the next stage, but you should not ask what possible solutions exist. Distribute your time properly Distribute your time properly Typically, an interview lasts 45 minutes, of which five are spent on greeting and a short introduction, and another five at the end on questions about the position you are interviewing for. There are 35 minutes left to solve the case. It’s important to distribute this time properly. From several user groups, it is better to choose one and focus on it. Then identify several problems that this group faces and choose one again. But it is better to offer several solutions. This way you will increase the probability that one of them will be good and will be able to demonstrate prioritization skills. When choosing one of the user groups and one problem, it is better to explain why you decided to focus on it. For example, it could be the largest group or the one that brings in the most revenue. Common mistakes Let me share some common mistakes that I've encountered in my practice. Jump straight to solutions Jump straight to solutions The most common mistake is to jump straight to sketching out ideas. A candidate is asked to improve a product, they open a website or application and starts listing ideas – add a filter here, and a button there. This is a 100% no hire, because it is unlikely that a candidate will be able to come up with something brilliant right away, without knowledge of the product, audience and business goals, rather he will demonstrate a lack of experience. Think that deep expertise in the product is necessary Think that deep expertise in the product is necessary In fact, it doesn't matter what kind of product you are creating or improving – a delivery application, a vending machine or Google search. In solving a case, it is important to show that you know the product approach and no matter what service you are working on, you know where to start. Follow templates too literally Follow templates too literally Once, I failed an interview because I followed a prepared template too literally instead of solving a case from my head. Solving a case requires not only formally going through all the steps from the template, but also the ability to rely on your own experience and adapt to changing task conditions on the fly. Therefore, it makes sense to solve several cases in advance to practice all the skills together. Conclusion It’s important to be prepared for all interviews, but I would pay special attention to the Product Sense Interview, as this is a key stage, where a product manager can demonstrate their skills. To be well-prepared, I would recommend reading Cracking PM Interview, watching videos of others going through the interview on YouTube and taking few mock interviews in advance. Also it’s important to prepare your own template for an answer and know some common mistakes that candidates often make. I hope this article helps you in your preparation, and wish you success in your interviews. I’m also curious about the challenges you’ve faced during the interview process, and what things helped you went through the product case interview.

Product Sense Interview: How to Prepare and Succeed

পণ্য সংবেদন সাক্ষাৎকার: কিভাবে প্রস্তুত এবং সফল

Product Sense-Interview: So bereiten Sie sich vor und sind erfolgreich

Entrevista sobre el producto: cómo prepararse y tener éxito

Entretien de Product Sense : comment se préparer et réussir

प्रोडक्ट सेंस इंटरव्यू: कैसे तैयारी करें और सफल हों

プロダクトセンスインタビュー：準備して成功する方法

제품 감각 인터뷰: 준비하고 성공하는 방법

Entrevista Product Sense: Como se preparar e ter sucesso

Интервью по продукту: как подготовиться и добиться успеха

Ürün Anlayışı Mülakatı: Nasıl Hazırlanılır ve Başarılı Olunur

Phỏng vấn Product Sense: Cách chuẩn bị và thành công

产品感知面试：如何准备并获得成功

Facebook. Where do I even begin with this colossal junkyard of a platform? It's like someone took the concept of social networking and doused it in a vat of incompetence. Let's talk about how Facebook has managed to mess up beyond all common sense. Firstly, their idea of content moderation seems to be more about randomly throwing darts at a board than actually using any sort of coherent logic. Legitimate posts in perfectly innocuous groups are being whisked away into the digital ether while spam proliferates like a disease. Seriously, how does a post about genuine advice or a thoughtful discussion get flagged and removed, but the umpteenth spammy post for shady sunglasses or miracle weight loss pills in totally irrelevant groups get a free pass? It's like Facebook has an algorithm that's not just broken but actively malicious. Facebook cares more about profit margins than the user experience. Instead of fostering a community where people can share meaningful ideas and connect on a genuine level, they've turned it into a virtual junkyard where the loudest, most obnoxious, spammy advertisers and paid-to-post parasites reign supreme. What happened to valuing quality over quantity? Now, it's all about how many eyeballs they can shove in front of spammy URL posts or advertisements, regardless of whether those eyeballs are seeing something worthwhile or utter garbage. It's a race to the bottom where meaningful content is drowned out by an endless barrage of clickbait. And let's not forget the privacy nightmares. Facebook has more tentacles in your personal life than an overbearing mother-in-law. They track your every move, every click, every like, and probably your wet dreams if they could. Remember when Facebook was caught red-handed with plans to sell user data and received support from Facebook’s most senior executives, including CEO Mark Zuckerberg and chief operating officer Sheryl Sandberg? Yeah, that happened, and somehow, we're all still here, feeding the beast with our personal information. According to approximately 4,000 pages of leaked internal documents obtained by NBC News, Zuckerberg allegedly supervised strategies aimed at centralizing the social network's influence and stifling competitors by using user data as leverage. This occurred despite public assurances from the company about protecting user data. The documents primarily cover the years 2011 to 2015. Facebook’s (Meta) latest spam joke Case in point: I joined a Facebook group dedicated to blogging and guest posting. Perfect, right? Wrong. I posted a meticulously crafted message with a URL, offering insightful tips on how to craft a perfect guest post. What did Facebook do? They promptly removed my post, citing some obscure community guideline violation. Meanwhile, the same group was inundated with spam posts about irrelevant topics, worthless websites, and other scammers, hawked by individuals paid to spread digital garbage like confetti at a clown convention. Facebook has lost its way. Digital Nonsense It's crystal clear that Facebook has lost its way. They seem more interested in padding their profits than ensuring a quality user experience. Maybe they're scrimping on competent programmers because their algorithms can't tell the difference between genuine content and spammy trash. There are better options out there, folks. Platforms that actually give a damn about quality and fairness. They know how to handle spam: swiftly and decisively, with bans and suspensions flying like justice from a vigilant sheriff. Meanwhile, Facebook twiddles its thumbs, letting the spammers run amok like unruly toddlers in a toy store. It's time for a change It's time for a change. Governments should consider banning this digital travesty of a platform. We deserve better. We deserve platforms that respect our time, our data, and our intelligence. Facebook had its chance. It's time to let it fade into irrelevance, like the distant memory of a regrettable haircut or Myspace. Facebook could have been a force for good - Instead, it’s a spam-loving machine The saddest part is that Facebook could have been a force for good. It could have been a platform where people from all walks of life could come together, share ideas, and foster genuine connections. Instead, it's a bloated, soulless husk of what it could have been—a place where spam rules and meaningful interaction is a rare gem buried under layers of digital detritus. Facebook is a joke Facebook is a joke—a sad, outdated relic in the age of superior social media alternatives. Let's move on, shall we? There's a whole world of digital connectivity out there, waiting for us to embrace platforms that actually value what matters. Facebook's love affair with scammers Facebook's love affair with scammers is a textbook example of corporate greed overshadowing ethical responsibility. They don't just turn a blind eye to scammers; they roll out the red carpet for them. It's like they have a special VIP section reserved for shady operators peddling everything from fake diplomas, get-rich-quick schemes, dodgy products and digital services. It's like they're saying, "Scam away, folks! Just make sure the ad revenue keeps flowing." They rake in cash hand over fist while unsuspecting users fall victim to schemes that would make even the most seasoned con artist blush. Facebook doesn't care. And the best part? Facebook doesn't care. They're too busy counting their ill-gotten gains to bother with pesky details like vetting advertisers or ensuring the legitimacy of promoted services. Who cares if someone's promoting a Ponzi scheme or selling snake oil disguised as miracle cures? Certainly not Facebook, as long as those sweet advertising dollars keep pouring in. Facebook's revenue - Willful ignorance Let's face it: a substantial chunk of Facebook's revenue likely comes from scammers. I mean, do they even bother to check? The obvious answer is no. It's the digital equivalent of renting out billboard space to a carnival barker without asking what's under the tent. They're complicit, whether through negligence or willful ignorance. So, while the rest of us navigate the internet with caution, Facebook gleefully lines its pockets with ad click money earned from the misfortune of others. It's a disgrace, a betrayal of trust, and a stark reminder that when it comes to ethics, Facebook is MIA. Facebook's laissez-faire attitude towards scammy and spammy advertisers is not just irresponsible—it's reprehensible. They owe their users better. They owe the digital community better. It’s time our governments finally put a stop to this mayhem. Hence why I have decided to start my own movement, a reporting agency that finds, responds to and reports every scam we come across. Then, provide a database every month to my national government that reflects the reported scams Meta refuses to remove after being reported. A visionary will emerge - Sooner than you think! Someday, a visionary will emerge—someone smarter, bolder, and more ethical than Zuckerberg and his cohort of profit-driven enablers. This visionary will revolutionize social media, leaving Facebook (or Meta, as they now call it) in the dust. (Watch this space) Picture this: a platform where authenticity reigns supreme. Only real people, genuine advertisers, and vetted content are allowed. No more scams, no more clickbait, no more shady dealings. It's a place where integrity is not just a buzzword but a guiding principle. This new social media titan will prioritize user safety and privacy above all else. Children will be protected, personal data will be guarded like a treasure, and transparency will be the cornerstone of every interaction. Imagine a digital space where users can connect, share, and thrive without fear of exploitation or deception. And yes, this platform will make money. But unlike Facebook, its success won't come at the expense of its users' trust. It will be a fair exchange where value is delivered without compromising principles. Advertising will be relevant and respectful, enhancing the user experience rather than detracting from it.  It’s coming, believe me. Maybe sooner than you think. Meta, beware. The winds of change are blowing, and it's only a matter of time until you become the next Myspace—a relic of a bygone era. The future belongs to those who innovate with integrity and prioritize the well-being of their community. So, brace yourself because the revolution is coming, and when it arrives, the world will finally have a social media platform worthy of its users. So, Facebook, if you're listening (which you probably are because, let's face it, you're always listening), get your act together. Clean up your act. Start valuing quality content and user experience over your bottom line driven by spam and scams. Stop treating us like commodities to be sold to the highest bidder. Until then, I'll be over here, trying to salvage what's left of my faith in social media. And maybe, just maybe, hoping for a platform that actually gives a damn about its legitimate users instead of spammers and scammers. It's time they cleaned up their act and started prioritizing integrity over profit, and governments start handing out major penalties Until then, caveat emptor: let the buyer beware, especially when it comes to anything Facebook promotes on its junkyard of depict, spam and filth. Facebook. Where do I even begin with this colossal junkyard of a platform? It's like someone took the concept of social networking and doused it in a vat of incompetence. Let's talk about how Facebook has managed to mess up beyond all common sense. Firstly, their idea of content moderation seems to be more about randomly throwing darts at a board than actually using any sort of coherent logic. Legitimate posts in perfectly innocuous groups are being whisked away into the digital ether while spam proliferates like a disease. Seriously, how does a post about genuine advice or a thoughtful discussion get flagged and removed, but the umpteenth spammy post for shady sunglasses or miracle weight loss pills in totally irrelevant groups get a free pass? It's like Facebook has an algorithm that's not just broken but actively malicious. Facebook cares more about profit margins than the user experience. Instead of fostering a community where people can share meaningful ideas and connect on a genuine level, they've turned it into a virtual junkyard where the loudest, most obnoxious, spammy advertisers and paid-to-post parasites reign supreme. What happened to valuing quality over quantity? Now, it's all about how many eyeballs they can shove in front of spammy URL posts or advertisements, regardless of whether those eyeballs are seeing something worthwhile or utter garbage. It's a race to the bottom where meaningful content is drowned out by an endless barrage of clickbait. And let's not forget the privacy nightmares. Facebook has more tentacles in your personal life than an overbearing mother-in-law. They track your every move, every click, every like, and probably your wet dreams if they could. Remember when Facebook was caught red-handed with plans to sell user data and received support from Facebook’s most senior executives, including CEO Mark Zuckerberg and chief operating officer Sheryl Sandberg? Yeah, that happened, and somehow, we're all still here, feeding the beast with our personal information. plans to sell user data According to approximately 4,000 pages of leaked internal documents obtained by NBC News, Zuckerberg allegedly supervised strategies aimed at centralizing the social network's influence and stifling competitors by using user data as leverage. This occurred despite public assurances from the company about protecting user data. The documents primarily cover the years 2011 to 2015. Facebook’s (Meta) latest spam joke Case in point: I joined a Facebook group dedicated to blogging and guest posting. Perfect, right? Wrong. I posted a meticulously crafted message with a URL, offering insightful tips on how to craft a perfect guest post . guest post What did Facebook do? What did Facebook do? They promptly removed my post, citing some obscure community guideline violation. Meanwhile, the same group was inundated with spam posts about irrelevant topics, worthless websites, and other scammers , hawked by individuals paid to spread digital garbage like confetti at a clown convention. spam posts other scammers Facebook has lost its way. Digital Nonsense It's crystal clear that Facebook has lost its way. They seem more interested in padding their profits than ensuring a quality user experience. Maybe they're scrimping on competent programmers because their algorithms can't tell the difference between genuine content and spammy trash. There are better options out there, folks. Platforms that actually give a damn about quality and fairness. They know how to handle spam: swiftly and decisively, with bans and suspensions flying like justice from a vigilant sheriff. Meanwhile, Facebook twiddles its thumbs, letting the spammers run amok like unruly toddlers in a toy store. It's time for a change It's time for a change. Governments should consider banning this digital travesty of a platform. We deserve better. We deserve platforms that respect our time, our data, and our intelligence. Facebook had its chance. It's time to let it fade into irrelevance, like the distant memory of a regrettable haircut or Myspace. Facebook could have been a force for good - Instead, it’s a spam-loving machine Facebook could have been a force for good - Instead, it’s a spam-loving machine The saddest part is that Facebook could have been a force for good. It could have been a platform where people from all walks of life could come together, share ideas, and foster genuine connections. Instead, it's a bloated, soulless husk of what it could have been—a place where spam rules and meaningful interaction is a rare gem buried under layers of digital detritus. Facebook Facebook is a joke Facebook is a joke—a sad, outdated relic in the age of superior social media alternatives. Let's move on, shall we? There's a whole world of digital connectivity out there, waiting for us to embrace platforms that actually value what matters. Facebook's love affair with scammers Facebook's love affair with scammers is a textbook example of corporate greed overshadowing ethical responsibility. They don't just turn a blind eye to scammers; they roll out the red carpet for them. It's like they have a special VIP section reserved for shady operators peddling everything from fake diplomas, get-rich-quick schemes, dodgy products and digital services. love affair with scammers shady operators fake It's like they're saying, "Scam away, folks! Just make sure the ad revenue keeps flowing." They rake in cash hand over fist while unsuspecting users fall victim to schemes that would make even the most seasoned con artist blush. Facebook doesn't care. And the best part? Facebook doesn't care. They're too busy counting their ill-gotten gains to bother with pesky details like vetting advertisers or ensuring the legitimacy of promoted services. Who cares if someone's promoting a Ponzi scheme or selling snake oil disguised as miracle cures? Certainly not Facebook, as long as those sweet advertising dollars keep pouring in. Facebook's revenue - Willful ignorance Let's face it: a substantial chunk of Facebook's revenue likely comes from scammers. I mean, do they even bother to check? The obvious answer is no. It's the digital equivalent of renting out billboard space to a carnival barker without asking what's under the tent. They're complicit, whether through negligence or willful ignorance. So, while the rest of us navigate the internet with caution, Facebook gleefully lines its pockets with ad click money earned from the misfortune of others. It's a disgrace, a betrayal of trust, and a stark reminder that when it comes to ethics, Facebook is MIA. Facebook's laissez-faire attitude towards scammy and spammy advertisers is not just irresponsible—it's reprehensible. They owe their users better. They owe the digital community better. It’s time our governments finally put a stop to this mayhem. Hence why I have decided to start my own movement, a reporting agency that finds, responds to and reports every scam we come across. Then, provide a database every month to my national government that reflects the reported scams Meta refuses to remove after being reported. the reported scams A visionary will emerge - Sooner than you think! Someday, a visionary will emerge—someone smarter, bolder, and more ethical than Zuckerberg and his cohort of profit-driven enablers. This visionary will revolutionize social media, leaving Facebook (or Meta, as they now call it) in the dust. (Watch this space) Picture this: a platform where authenticity reigns supreme. Only real people, genuine advertisers, and vetted content are allowed. No more scams, no more clickbait, no more shady dealings. It's a place where integrity is not just a buzzword but a guiding principle. authenticity content This new social media titan will prioritize user safety and privacy above all else. Children will be protected, personal data will be guarded like a treasure, and transparency will be the cornerstone of every interaction. Imagine a digital space where users can connect, share, and thrive without fear of exploitation or deception. And yes, this platform will make money. But unlike Facebook, its success won't come at the expense of its users' trust. It will be a fair exchange where value is delivered without compromising principles. Advertising will be relevant and respectful, enhancing the user experience rather than detracting from it.  It’s coming, believe me. Maybe sooner than you think. Meta, beware. The winds of change are blowing, and it's only a matter of time until you become the next Myspace—a relic of a bygone era. The future belongs to those who innovate with integrity and prioritize the well-being of their community. So, brace yourself because the revolution is coming, and when it arrives, the world will finally have a social media platform worthy of its users. So, Facebook, if you're listening (which you probably are because, let's face it, you're always listening), get your act together. Clean up your act. Start valuing quality content and user experience over your bottom line driven by spam and scams. Stop treating us like commodities to be sold to the highest bidder. Until then, I'll be over here, trying to salvage what's left of my faith in social media. And maybe, just maybe, hoping for a platform that actually gives a damn about its legitimate users instead of spammers and scammers. It's time they cleaned up their act and started prioritizing integrity over profit, and governments start handing out major penalties Until then, caveat emptor: let the buyer beware, especially when it comes to anything Facebook promotes on its junkyard of depict, spam and filth.

Facebook Has Managed To Mess Up Beyond All Common Sense. A Social Media Nightmare 

From Nuisance to News Sense: Conclusion, Limitations and Ethic, and References

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Jeremiah Milbauer, Carnegie Mellon University, Pittsburgh PA, USA (email: {jmilbaue | sherryw}@cs.cmu.edu);
(2) Ziqi Ding, Carnegie Mellon University, Pittsburgh PA, USA (e-mail: {ziqiding | zhijinw}@andrew.cmu.edu)
(3) Tongshuang Wu, Carnegie Mellon University, Pittsburgh PA, USA. Table of Links Abstract and Introduction
Related Work
The NEWSSENSE Framework
Pilot Study
System Overview
Discussion
Conclusion, Limitations and Ethics, and References
A. Appendix: Prompt for Claim Extraction A. Appendix: Prompt for Claim Extraction This is the prompt used for claim extraction from news article sentences: Extract all the claims from a sentence, ignoring extraneous words such as unimportant adverbs. A sentence may contain multiple claims. Each claim should be of the form < subject> <predicate> <object>, and should have the first occurrence of any pronouns replaced by their antecedents. Sentence: "The 3rd and 4th stations all announced that they would be postponed, and the Monaco station was subsequently cancelled." Claim: Monaco station was cancelled. Claim: 4th stations announced they would be postponed. Claim: The 3rd stations announced they would be postponed. Claim: The 4th stations postponed. Claim: The 3rd stations postponed. Sentence: "Lewis Hamilton and Mercedes have once again confirmed themselves as drivers and constructors world champions." Claim: Mercedes confirmed themselves as constructors world champions. Claim: Lewis Hamilton confirmed themselves as drivers world champions. Sentence: "Local organizers in East Palestine, Ohio on Monday said their activism has successfully pressured rail company Norfolk Southern to agree to a limited relocation plan for some residents affected by last month’s train derailment, but added they have no intention of backing down from their demand for justice for thousands of people in the area who are struggling in the aftermath of the accident." Claim: Local organizers said their activism has pressured rail company Norfolk Southern to agree to a limited relocation plan. Claim: Local organizers have no intention of backing down from their demand for justice. Claim: Rail company Norfolk Southern agree to a limited relocation plan. Sentence: This paper is available on arxiv under CC 4.0 license. Authors: (1) Jeremiah Milbauer, Carnegie Mellon University, Pittsburgh PA, USA (email: {jmilbaue | sherryw}@cs.cmu.edu); (2) Ziqi Ding, Carnegie Mellon University, Pittsburgh PA, USA (e-mail: {ziqiding | zhijinw}@andrew.cmu.edu) (3) Tongshuang Wu, Carnegie Mellon University, Pittsburgh PA, USA. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Jeremiah Milbauer, Carnegie Mellon University, Pittsburgh PA, USA (email: {jmilbaue | sherryw}@cs.cmu.edu); (2) Ziqi Ding, Carnegie Mellon University, Pittsburgh PA, USA (e-mail: {ziqiding | zhijinw}@andrew.cmu.edu) (3) Tongshuang Wu, Carnegie Mellon University, Pittsburgh PA, USA. Table of Links Abstract and Introduction Related Work The NEWSSENSE Framework Pilot Study System Overview Discussion Conclusion, Limitations and Ethics, and References A. Appendix: Prompt for Claim Extraction Abstract and Introduction Abstract and Introduction Related Work Related Work The NEWSSENSE Framework The NEWSSENSE Framework Pilot Study Pilot Study System Overview System Overview Discussion Discussion Conclusion, Limitations and Ethics, and References Conclusion, Limitations and Ethics, and References A. Appendix: Prompt for Claim Extraction A. Appendix: Prompt for Claim Extraction A. Appendix: Prompt for Claim Extraction This is the prompt used for claim extraction from news article sentences: Extract all the claims from a sentence, ignoring extraneous words such as unimportant adverbs. A sentence may contain multiple claims. Each claim should be of the form < subject> <predicate> <object>, and should have the first occurrence of any pronouns replaced by their antecedents. Sentence: "The 3rd and 4th stations all announced that they would be postponed, and the Monaco station was subsequently cancelled." Claim: Monaco station was cancelled. Claim: 4th stations announced they would be postponed. Claim: The 3rd stations announced they would be postponed. Claim: The 4th stations postponed. Claim: The 3rd stations postponed. Sentence: "Lewis Hamilton and Mercedes have once again confirmed themselves as drivers and constructors world champions." Claim: Mercedes confirmed themselves as constructors world champions. Claim: Lewis Hamilton confirmed themselves as drivers world champions. Sentence: "Local organizers in East Palestine, Ohio on Monday said their activism has successfully pressured rail company Norfolk Southern to agree to a limited relocation plan for some residents affected by last month’s train derailment, but added they have no intention of backing down from their demand for justice for thousands of people in the area who are struggling in the aftermath of the accident." Claim: Local organizers said their activism has pressured rail company Norfolk Southern to agree to a limited relocation plan. Claim: Local organizers have no intention of backing down from their demand for justice. Claim: Rail company Norfolk Southern agree to a limited relocation plan. Sentence:

From Nuisance to News Sense: Appendix: Prompt for Claim Extraction

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Jeremiah Milbauer, Carnegie Mellon University, Pittsburgh PA, USA (email: {jmilbaue | sherryw}@cs.cmu.edu);
(2) Ziqi Ding, Carnegie Mellon University, Pittsburgh PA, USA (e-mail: {ziqiding | zhijinw}@andrew.cmu.edu)
(3) Tongshuang Wu, Carnegie Mellon University, Pittsburgh PA, USA. Table of Links Abstract and Introduction
Related Work
The NEWSSENSE Framework
Pilot Study
System Overview
Discussion
Conclusion, Limitations and Ethics, and References
A. Appendix: Prompt for Claim Extraction 6 Discussion NEWSSENSE provides an intuitive and effective interface for integrating information from a large cluster of news articles into a single, focused reading experience. Although applied in this demo to news articles, the NEWSSENSE framework could just as easily be applied to the analysis of other types of document clusters as well. The pipeline itself is highly modular, and can easily adopt advancements in NLP technologies to increase the accuracy or decrease processing time. 6.1 Future Work The generality of the NEWSSENSE also introduces a number of opportunities for future development. Expanding the Scope of NewsReader Often, articles contain references to past events. In the future, we would like to explore the possibility of extending the NEWSSENSE framework beyond the immediately article clusters to include all relevant articles in a timeline of events. Additionally, as we explored the NEWSSENSE framework, we noticed that the clustering approach we used – the Google News Stories – sometimes established associations between source news articles, and background primary source articles. As a result, we would encourage further exploration of the NewsSense framework applied to heterogenous and primary-source document collections, whcih might include primary scholarly literature. NLP Pipeline Improvements Because the NEWSSENSE pipeline is modular, a number of improvements can be explored. For sentence segmentation, methods that use a fine-tuned language model could improve the speed of segmentation. For sentence filtering, we relied on a pretrained sentence retrieval model – but future iterations of NEWSSENSE could use a sentence retrieval model fine-tuned on “unrelated" pairs of MNLI sentences. We also found that in many cases, the NLI algorithm used for claim linking made mistakes, perhaps owing to the fact that news articles may not perfectly match models trained on MNLI. Other NLI approaches could be explored, such as SeNtLI (Schuster et al., 2022) (which is designed to work for both individual premises and longer sentences) and LAIT (Milbauer et al., 2023), which speeds up inference time through late interaction. More Useful Information Our final version of NEWSSENSE focused on a relatively paired-down and streamlined interface. However, users did suggest that they would like to see article summaries, and we identified that in many cases key information is repeated across multiple articles. We would consider adding a way for NEWSSENSE to convey the highlights – key claims from across the article cluster – when a user is reading an article. We noticed other forms of unintended but incredibly useful functionality: For example, as stories develop, new facts emerge that may contradict old ones. This means that newer articles might supersede older ones. Future iterations of NEWSSENSE should help readers understand when a contradiction may be due to evolving stories. Deployment A larger-scale user study would help determine what further improvements could be made to the framework. Our fully interactive interface would help us run a study at larger scale. This paper is available on arxiv under CC 4.0 license. Authors: (1) Jeremiah Milbauer, Carnegie Mellon University, Pittsburgh PA, USA (email: {jmilbaue | sherryw}@cs.cmu.edu); (2) Ziqi Ding, Carnegie Mellon University, Pittsburgh PA, USA (e-mail: {ziqiding | zhijinw}@andrew.cmu.edu) (3) Tongshuang Wu, Carnegie Mellon University, Pittsburgh PA, USA. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Jeremiah Milbauer, Carnegie Mellon University, Pittsburgh PA, USA (email: {jmilbaue | sherryw}@cs.cmu.edu); (2) Ziqi Ding, Carnegie Mellon University, Pittsburgh PA, USA (e-mail: {ziqiding | zhijinw}@andrew.cmu.edu) (3) Tongshuang Wu, Carnegie Mellon University, Pittsburgh PA, USA. Table of Links Abstract and Introduction Related Work The NEWSSENSE Framework Pilot Study System Overview Discussion Conclusion, Limitations and Ethics, and References A. Appendix: Prompt for Claim Extraction Abstract and Introduction Abstract and Introduction Related Work Related Work The NEWSSENSE Framework The NEWSSENSE Framework Pilot Study Pilot Study System Overview System Overview Discussion Discussion Conclusion, Limitations and Ethics, and References Conclusion, Limitations and Ethics, and References A. Appendix: Prompt for Claim Extraction A. Appendix: Prompt for Claim Extraction 6 Discussion NEWSSENSE provides an intuitive and effective interface for integrating information from a large cluster of news articles into a single, focused reading experience. Although applied in this demo to news articles, the NEWSSENSE framework could just as easily be applied to the analysis of other types of document clusters as well. The pipeline itself is highly modular, and can easily adopt advancements in NLP technologies to increase the accuracy or decrease processing time. 6.1 Future Work The generality of the NEWSSENSE also introduces a number of opportunities for future development. Expanding the Scope of NewsReader Often, articles contain references to past events. In the future, we would like to explore the possibility of extending the NEWSSENSE framework beyond the immediately article clusters to include all relevant articles in a timeline of events. Expanding the Scope of NewsReader Additionally, as we explored the NEWSSENSE framework, we noticed that the clustering approach we used – the Google News Stories – sometimes established associations between source news articles, and background primary source articles. As a result, we would encourage further exploration of the NewsSense framework applied to heterogenous and primary-source document collections, whcih might include primary scholarly literature. news primary source NewsSense NLP Pipeline Improvements Because the NEWSSENSE pipeline is modular, a number of improvements can be explored. For sentence segmentation, methods that use a fine-tuned language model could improve the speed of segmentation. For sentence filtering, we relied on a pretrained sentence retrieval model – but future iterations of NEWSSENSE could use a sentence retrieval model fine-tuned on “unrelated" pairs of MNLI sentences. We also found that in many cases, the NLI algorithm used for claim linking made mistakes, perhaps owing to the fact that news articles may not perfectly match models trained on MNLI. Other NLI approaches could be explored, such as SeNtLI (Schuster et al., 2022) (which is designed to work for both individual premises and longer sentences) and LAIT (Milbauer et al., 2023), which speeds up inference time through late interaction. NLP Pipeline Improvements More Useful Information Our final version of NEWSSENSE focused on a relatively paired-down and streamlined interface. However, users did suggest that they would like to see article summaries, and we identified that in many cases key information is repeated across multiple articles. We would consider adding a way for NEWSSENSE to convey the highlights – key claims from across the article cluster – when a user is reading an article. We noticed other forms of unintended but incredibly useful functionality: For example, as stories develop, new facts emerge that may contradict old ones. This means that newer articles might supersede older ones. Future iterations of NEWSSENSE should help readers understand when a contradiction may be due to evolving stories. More Useful Information Deployment A larger-scale user study would help determine what further improvements could be made to the framework. Our fully interactive interface would help us run a study at larger scale. Deployment

From Nuisance to News Sense: Discussion

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Jeremiah Milbauer, Carnegie Mellon University, Pittsburgh PA, USA (email: {jmilbaue | sherryw}@cs.cmu.edu);
(2) Ziqi Ding, Carnegie Mellon University, Pittsburgh PA, USA (e-mail: {ziqiding | zhijinw}@andrew.cmu.edu)
(3) Tongshuang Wu, Carnegie Mellon University, Pittsburgh PA, USA. Table of Links Abstract and Introduction
Related Work
The NEWSSENSE Framework
Pilot Study
System Overview
Discussion
Conclusion, Limitations and Ethics, and References
A. Appendix: Prompt for Claim Extraction 3 The NEWSSENSE Framework The core philosophy behind NEWSSENSE is to go beyond article aggregation by integrating the information contained within a cluster of related news articles into the reading experience. NEWSSENSE starts with a single “focus" article and a set of related “background" articles. The distinction between focus and background article is arbitrary, as any article within the cluster could be designated the focus article. NEWSSENSE then identifies claims within the focus article that are related – either by contradiction or entailment – to claims within the background articles. The claims in the focus article are then highlighted, and linked to the background articles so that users can explore the supporting or contradicting evidence for a given claim without just relying on third-party measurements of bias or credibility. The NEWSSENSE interface has three primary components: a Focus Article, Sentence Highlights, and External Evidence. Together, these elements display the computed connections between the focus article and the background articles. 3.1 Focus Article The NEWSSENSE interface features a central panel that displays the focus article, including the entire news article the reader is interested in primarily. The focus article can be presented through a dedicated application, or by adding NEWSREADER as an overlay on top of the existing web browsing experience. 3.2 Sentence Highlights The interface highlights claims made in the focus article that have supporting articles in green underlines, while claims with contradicting articles are highlighted in red underlines. By doing so, readers can quickly and easily identify areas of agreement and disagreement across different news sources. 3.3 External Evidence When the reader hovers or clicks on a highlighted claim, an overlay panel appears, containing the supporting or contradicting claim excerpts, as well as their news sources. Links can be annotated with information about the general credibility or political slant of the referenced news venue. For readers’ convenience, each supporting or contradicting claim is clickable and directs the reader to origin of the associated claim. This allows readers to quickly access the relevant claims without having to search through an entire article. For a standalone NEWSSENSE interface, readers will be prompted with a “back” button in the secondary articles to quickly go back to the focus article. By providing this functionality, readers can easily navigate through the focus and secondary articles and compare viewpoints, further enhancing their understanding of the news story. This paper is available on arxiv under CC 4.0 license. Authors: (1) Jeremiah Milbauer, Carnegie Mellon University, Pittsburgh PA, USA (email: {jmilbaue | sherryw}@cs.cmu.edu); (2) Ziqi Ding, Carnegie Mellon University, Pittsburgh PA, USA (e-mail: {ziqiding | zhijinw}@andrew.cmu.edu) (3) Tongshuang Wu, Carnegie Mellon University, Pittsburgh PA, USA. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Jeremiah Milbauer, Carnegie Mellon University, Pittsburgh PA, USA (email: {jmilbaue | sherryw}@cs.cmu.edu); (2) Ziqi Ding, Carnegie Mellon University, Pittsburgh PA, USA (e-mail: {ziqiding | zhijinw}@andrew.cmu.edu) (3) Tongshuang Wu, Carnegie Mellon University, Pittsburgh PA, USA. Table of Links Abstract and Introduction Related Work The NEWSSENSE Framework Pilot Study System Overview Discussion Conclusion, Limitations and Ethics, and References A. Appendix: Prompt for Claim Extraction Abstract and Introduction Abstract and Introduction Related Work Related Work The NEWSSENSE Framework The NEWSSENSE Framework Pilot Study Pilot Study System Overview System Overview Discussion Discussion Conclusion, Limitations and Ethics, and References Conclusion, Limitations and Ethics, and References A. Appendix: Prompt for Claim Extraction A. Appendix: Prompt for Claim Extraction 3 The NEWSSENSE Framework The core philosophy behind NEWSSENSE is to go beyond article aggregation by integrating the information contained within a cluster of related news articles into the reading experience. NEWSSENSE starts with a single “focus" article and a set of related “background" articles. The distinction between focus and background article is arbitrary, as any article within the cluster could be designated the focus article. NEWSSENSE then identifies claims within the focus article that are related – either by contradiction or entailment – to claims within the background articles. The claims in the focus article are then highlighted, and linked to the background articles so that users can explore the supporting or contradicting evidence for a given claim without just relying on third-party measurements of bias or credibility. The NEWSSENSE interface has three primary components: a Focus Article, Sentence Highlights, and External Evidence. Together, these elements display the computed connections between the focus article and the background articles. 3.1 Focus Article The NEWSSENSE interface features a central panel that displays the focus article, including the entire news article the reader is interested in primarily. The focus article can be presented through a dedicated application, or by adding NEWSREADER as an overlay on top of the existing web browsing experience. 3.2 Sentence Highlights The interface highlights claims made in the focus article that have supporting articles in green underlines, while claims with contradicting articles are highlighted in red underlines. By doing so, readers can quickly and easily identify areas of agreement and disagreement across different news sources. 3.3 External Evidence When the reader hovers or clicks on a highlighted claim, an overlay panel appears, containing the supporting or contradicting claim excerpts, as well as their news sources. Links can be annotated with information about the general credibility or political slant of the referenced news venue. For readers’ convenience, each supporting or contradicting claim is clickable and directs the reader to origin of the associated claim. This allows readers to quickly access the relevant claims without having to search through an entire article. For a standalone NEWSSENSE interface, readers will be prompted with a “back” button in the secondary articles to quickly go back to the focus article. By providing this functionality, readers can easily navigate through the focus and secondary articles and compare viewpoints, further enhancing their understanding of the news story.

From Nuisance to News Sense: The NEWSSENSE Framework

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Jeremiah Milbauer, Carnegie Mellon University, Pittsburgh PA, USA (email: {jmilbaue | sherryw}@cs.cmu.edu);
(2) Ziqi Ding, Carnegie Mellon University, Pittsburgh PA, USA (e-mail: {ziqiding | zhijinw}@andrew.cmu.edu)
(3) Tongshuang Wu, Carnegie Mellon University, Pittsburgh PA, USA. Table of Links Abstract and Introduction
Related Work
The NEWSSENSE Framework
Pilot Study
System Overview
Discussion
Conclusion, Limitations and Ethics, and References
A. Appendix: Prompt for Claim Extraction 2 Related Work This section covers related research across media analytics, sensemaking, and natural language processing. Though some core ideas of this work have been explored in the past, to our knowledge they have never been combined in a single system. Media Bias and Analytics Research on media bias includes academic research to study social media sharing patterns (Roberts et al., 2021; Bakshy et al., 2015) and bias within media publications (Flaxman et al., 2016; Hamborg et al., 2019; Groseclose and Milyo, 2005). Commercial products exist in this area as well, such as the media bias charts of AllSides [5], which classifies political slant into one of five categories, and Ad Fontes Media [6], which models both political slant and factual credibility. Research on news and social content aggregation has focused primarily on headline detection, timeline construction and clustering (Bouras and Tsogkas, 2012; Laban and Hearst, 2017), and event detection (Atefeh and Khreich, 2015; Kumaran and Allan, 2004). There exist user-oriented products in this space, such as Google News Stories [7], and Ground.news [8]. Some outlets, such as Propublica, aggregate their news stories into timelines [9]. Reading Interfaces and Sensemaking Recent work on reading interfaces has primarily focused on scientific literature, augmenting documents with information about cited papers (Lo et al., 2023; Kang et al., 2022), or augmenting references within the documents themselves (Head et al., 2021). For the News domain specifically, Laban and Hearst (2017) aggregates articles and extracts key quotes to construct a timeline for a given story. We are also aware of an abstract describing work to combine multiple article headlines and ledes into a single digestible form, though no follow-up is available (Glassman et al., 2020). Fact Verification and NLI Natural Language Inference is a task focused on classifying the relationship between a pair of sentences as either “neutral", “entailment", or “contradiction." Datasets such as SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2017) have become major benchmarks for natural language processing research. Recent work has also considered document-level NLI (Koreeda and Manning, 2021; Chen et al., 2022), as well as cross-document reasoning based in NLI (Schuster et al., 2022), and scalable pairwise reasoning (Milbauer et al., 2023). There is also a growing body of work on NLP systems for fact verification and attribution. Recent datasets include FEVER (Thorne et al., 2018) and VitaminC (Schuster et al., 2021), as well as datasets focused on real-world examples of updating, editing, and citing claims in domains like news and Wikipedia (Petroni et al., 2022; Spangher et al., 2022; Iv et al., 2022). [5] https://www.allsides.com/media-bias/media-bias-ratingmethods [6] https://adfontesmedia.com/interactive-media-bias-chart/ [7] https://news.google.com [8] https://ground.news/ [9] https://www.propublica.org/series This paper is available on arxiv under CC 4.0 license. Authors: (1) Jeremiah Milbauer, Carnegie Mellon University, Pittsburgh PA, USA (email: {jmilbaue | sherryw}@cs.cmu.edu); (2) Ziqi Ding, Carnegie Mellon University, Pittsburgh PA, USA (e-mail: {ziqiding | zhijinw}@andrew.cmu.edu) (3) Tongshuang Wu, Carnegie Mellon University, Pittsburgh PA, USA. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Jeremiah Milbauer, Carnegie Mellon University, Pittsburgh PA, USA (email: {jmilbaue | sherryw}@cs.cmu.edu); (2) Ziqi Ding, Carnegie Mellon University, Pittsburgh PA, USA (e-mail: {ziqiding | zhijinw}@andrew.cmu.edu) (3) Tongshuang Wu, Carnegie Mellon University, Pittsburgh PA, USA. Table of Links Abstract and Introduction Related Work The NEWSSENSE Framework Pilot Study System Overview Discussion Conclusion, Limitations and Ethics, and References A. Appendix: Prompt for Claim Extraction Abstract and Introduction Abstract and Introduction Related Work Related Work The NEWSSENSE Framework The NEWSSENSE Framework Pilot Study Pilot Study System Overview System Overview Discussion Discussion Conclusion, Limitations and Ethics, and References Conclusion, Limitations and Ethics, and References A. Appendix: Prompt for Claim Extraction A. Appendix: Prompt for Claim Extraction 2 Related Work This section covers related research across media analytics, sensemaking, and natural language processing. Though some core ideas of this work have been explored in the past, to our knowledge they have never been combined in a single system. Media Bias and Analytics Research on media bias includes academic research to study social media sharing patterns (Roberts et al., 2021; Bakshy et al., 2015) and bias within media publications (Flaxman et al., 2016; Hamborg et al., 2019; Groseclose and Milyo, 2005). Commercial products exist in this area as well, such as the media bias charts of AllSides [5], which classifies political slant into one of five categories, and Ad Fontes Media [6], which models both political slant and factual credibility. Research on news and social content aggregation has focused primarily on headline detection, timeline construction and clustering (Bouras and Tsogkas, 2012; Laban and Hearst, 2017), and event detection (Atefeh and Khreich, 2015; Kumaran and Allan, 2004). There exist user-oriented products in this space, such as Google News Stories [7], and Ground.news [8]. Some outlets, such as Propublica, aggregate their news stories into timelines [9]. Reading Interfaces and Sensemaking Recent work on reading interfaces has primarily focused on scientific literature, augmenting documents with information about cited papers (Lo et al., 2023; Kang et al., 2022), or augmenting references within the documents themselves (Head et al., 2021). Reading Interfaces and Sensemaking For the News domain specifically, Laban and Hearst (2017) aggregates articles and extracts key quotes to construct a timeline for a given story. We are also aware of an abstract describing work to combine multiple article headlines and ledes into a single digestible form, though no follow-up is available (Glassman et al., 2020). Fact Verification and NLI Natural Language Inference is a task focused on classifying the relationship between a pair of sentences as either “neutral", “entailment", or “contradiction." Datasets such as SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2017) have become major benchmarks for natural language processing research. Recent work has also considered document-level NLI (Koreeda and Manning, 2021; Chen et al., 2022), as well as cross-document reasoning based in NLI (Schuster et al., 2022), and scalable pairwise reasoning (Milbauer et al., 2023). Fact Verification and NLI There is also a growing body of work on NLP systems for fact verification and attribution. Recent datasets include FEVER (Thorne et al., 2018) and VitaminC (Schuster et al., 2021), as well as datasets focused on real-world examples of updating, editing, and citing claims in domains like news and Wikipedia (Petroni et al., 2022; Spangher et al., 2022; Iv et al., 2022). [5] https://www.allsides.com/media-bias/media-bias-ratingmethods [6] https://adfontesmedia.com/interactive-media-bias-chart/ [7] https://news.google.com [8] https://ground.news/ [9] https://www.propublica.org/series

From Nuisance to News Sense: Related Work

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Jeremiah Milbauer, Carnegie Mellon University, Pittsburgh PA, USA (email: {jmilbaue | sherryw}@cs.cmu.edu);
(2) Ziqi Ding, Carnegie Mellon University, Pittsburgh PA, USA (e-mail: {ziqiding | zhijinw}@andrew.cmu.edu)
(3) Tongshuang Wu, Carnegie Mellon University, Pittsburgh PA, USA. Table of Links Abstract and Introduction
Related Work
The NEWSSENSE Framework
Pilot Study
System Overview
Discussion
Conclusion, Limitations and Ethics, and References
A. Appendix: Prompt for Claim Extraction Abstract Reading and understanding the stories in the news is increasingly difficult. Reporting on stories evolves rapidly, politicized news venues offer different perspectives (and sometimes different facts), and misinformation is rampant. However, Existing solutions merely aggregate an overwhelming amount of information from heterogenous sources, such as different news outlets, social media, and news bias rating agencies. We present NEWSSENSE, a novel sensemaking tool and reading interface designed to collect and integrate information from multiple news articles on a central topic. NEWSSENSE augments a central, grounding article of the user’s choice by linking it to related articles from different sources, providing inline highlights on how specific claims in the chosen article are either supported or contradicted by information from other articles. Using NEWSSENSE, users can seamlessly digest and cross-check multiple information sources without disturbing their natural reading flow. Our pilot study shows that NewsSense has the potential to help users identify key information, verify the credibility of news articles, and explore different perspectives. We opensource NewsSense at github.com/jmilbauer/NewsSense, and a demo video is hosted at youtu.be/2D5LYbsQJak. 1 Introduction Why is it so hard, and so exhausting, to read the news? In the quest for knowledge, news readers today must contend with a rapidly evolving 24-hour news cycle, multiple news venues competing for attention and clicks, and the challenge of integrating fact-based reporting, opinion pieces, and social media commentary (Lazer et al., 2018; Benkler et al., 2018; Farkas and Schou, 2019). With news becoming increasingly politicized (Faris et al., 2017) readers also face the challenge of identifying and avoiding misinformation, disinformation, and hyperbolic “clickbait" as they try to remain informed about the world around them. Various solutions have been proposed to assist with users’ news reading. For example, media watchdog companies have created media bias charts to represent political leaning and credibility of news sources [1] [2]. However, these resources force users to rely on potentially untrustworthy third-party designations of media bias, which treat each news source as a whole, without digging into specific articles or topics. While novel automatic fact checking (Thorne et al., 2018) and fake news detection (Zhou and Zafarani, 2020; Chen et al., 2015) systems can provide verification per-article, these approaches typically rely on a preordained corpus of verified facts which cannot keep up with the always-evolving facts, and may not reflect user preferences or multilateral perspectives. Aggregating articles from heterogeneous sources seem a more promising direction for cross-checking new facts (without predetermined groundtruths) and collecting different perspectives, but existing attempts are still too coarse and overwhelming. For example, both Google News’ “Stories" [3] feature and Ground.news [4] collect articles about the same events but display them in the form of exhaustive lists – Users are still forced to read and compare each article on its own. We argue that instead of simply collecting and aggregating news articles, information and claims from multiple sources should be integrated in a way that allows users to identify fine-grained claim-level bias, spin, controversy, or evidence. We present NEWSSENSE, a novel framework for sensemaking within a cluster of documents, to address the three key problems of news reading – bias, factuality, and article overload – in a single streamlined interface. NEWSSENSE leverages existing modular NLP techniques to identify and link claims made across a cluster of news articles, such that these articles become references for each other. NEWSSENSE also displays the linking information using an interactive reading interface, which allows users to easily explore the cross-document connections without being overwhelmed. In pilot user studies, we see that the NEWSSENSE framework has the potential to help users identify key information, verify the credibility of news articles, and explore different perspectives. While NEWSSENSE is primarily implemented for news articles, our framework can be easily generalized to other assisted reading and cross-checking scenarios (e.g., compare multiple manuscripts in literature reviews). The key contributions of NEWSSENSE are: A pipeline for analyzing the connections between a collection of documents.


A two-stage method for efficiently computing cross-document links between claims that support or contradict each other, enabling “reference-free” fact checking.


A framework for visualizing cross-document connections, and integrating claims from multiple documents into a single reading experience. We conclude by discussing the generality and potential social benefits of NEWSSENSE. [1] https://www.allsides.com/media-bias/media-bias-ratingmethods [2] https://adfontesmedia.com/interactive-media-bias-chart/ [3] https://news.google.com/stories/ [4] https://ground.news This paper is available on arxiv under CC 4.0 license. Authors: (1) Jeremiah Milbauer, Carnegie Mellon University, Pittsburgh PA, USA (email: {jmilbaue | sherryw}@cs.cmu.edu); (2) Ziqi Ding, Carnegie Mellon University, Pittsburgh PA, USA (e-mail: {ziqiding | zhijinw}@andrew.cmu.edu) (3) Tongshuang Wu, Carnegie Mellon University, Pittsburgh PA, USA. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Jeremiah Milbauer, Carnegie Mellon University, Pittsburgh PA, USA (email: {jmilbaue | sherryw}@cs.cmu.edu); (2) Ziqi Ding, Carnegie Mellon University, Pittsburgh PA, USA (e-mail: {ziqiding | zhijinw}@andrew.cmu.edu) (3) Tongshuang Wu, Carnegie Mellon University, Pittsburgh PA, USA. Table of Links Abstract and Introduction Related Work The NEWSSENSE Framework Pilot Study System Overview Discussion Conclusion, Limitations and Ethics, and References A. Appendix: Prompt for Claim Extraction Abstract and Introduction Abstract and Introduction Related Work Related Work The NEWSSENSE Framework The NEWSSENSE Framework Pilot Study Pilot Study System Overview System Overview Discussion Discussion Conclusion, Limitations and Ethics, and References Conclusion, Limitations and Ethics, and References A. Appendix: Prompt for Claim Extraction A. Appendix: Prompt for Claim Extraction Abstract Reading and understanding the stories in the news is increasingly difficult. Reporting on stories evolves rapidly, politicized news venues offer different perspectives (and sometimes different facts), and misinformation is rampant. However, Existing solutions merely aggregate an overwhelming amount of information from heterogenous sources, such as different news outlets, social media, and news bias rating agencies. We present NEWSSENSE, a novel sensemaking tool and reading interface designed to collect and integrate information from multiple news articles on a central topic. NEWSSENSE augments a central, grounding article of the user’s choice by linking it to related articles from different sources, providing inline highlights on how specific claims in the chosen article are either supported or contradicted by information from other articles. Using NEWSSENSE, users can seamlessly digest and cross-check multiple information sources without disturbing their natural reading flow. Our pilot study shows that NewsSense has the potential to help users identify key information, verify the credibility of news articles, and explore different perspectives. We opensource NewsSense at github.com/jmilbauer/NewsSense, and a demo video is hosted at youtu.be/2D5LYbsQJak. aggregate integrate 1 Introduction Why is it so hard, and so exhausting, to read the news? In the quest for knowledge, news readers today must contend with a rapidly evolving 24-hour news cycle, multiple news venues competing for attention and clicks, and the challenge of integrating fact-based reporting, opinion pieces, and social media commentary (Lazer et al., 2018; Benkler et al., 2018; Farkas and Schou, 2019). With news becoming increasingly politicized (Faris et al., 2017) readers also face the challenge of identifying and avoiding misinformation, disinformation, and hyperbolic “clickbait" as they try to remain informed about the world around them. Various solutions have been proposed to assist with users’ news reading. For example, media watchdog companies have created media bias charts to represent political leaning and credibility of news sources [1] [2]. However, these resources force users to rely on potentially untrustworthy third-party designations of media bias, which treat each news source as a whole, without digging into specific articles or topics. news source While novel automatic fact checking (Thorne et al., 2018) and fake news detection (Zhou and Zafarani, 2020; Chen et al., 2015) systems can provide verification per-article, these approaches typically rely on a preordained corpus of verified facts which cannot keep up with the always-evolving facts, and may not reflect user preferences or multilateral perspectives. Aggregating articles from heterogeneous sources seem a more promising direction for cross-checking new facts (without predetermined groundtruths) and collecting different perspectives, but existing attempts are still too coarse and overwhelming. For example, both Google News’ “Stories" [3] feature and Ground.news [4] collect articles about the same events but display them in the form of exhaustive lists – Users are still forced to read and compare each article on its own. We argue that instead of simply collecting and aggregating news articles, information and claims from multiple sources should be integrated in a way that allows users to identify fine-grained claim-level bias, spin, controversy, or evidence. integrated We present NEWSSENSE, a novel framework for sensemaking within a cluster of documents, to address the three key problems of news reading – bias, factuality, and article overload – in a single streamlined interface. NEWSSENSE leverages existing modular NLP techniques to identify and link claims made across a cluster of news articles , such that these articles become references for each other. NEWSSENSE also displays the linking information using an interactive reading interface , which allows users to easily explore the cross-document connections without being overwhelmed. In pilot user studies, we see that the NEWSSENSE framework has the potential to help users identify key information, verify the credibility of news articles, and explore different perspectives. a cluster of news articles interactive reading interface While NEWSSENSE is primarily implemented for news articles, our framework can be easily generalized to other assisted reading and cross-checking scenarios (e.g., compare multiple manuscripts in literature reviews). The key contributions of NEWSSENSE are: A pipeline for analyzing the connections between a collection of documents. A two-stage method for efficiently computing cross-document links between claims that support or contradict each other, enabling “reference-free” fact checking. A framework for visualizing cross-document connections, and integrating claims from multiple documents into a single reading experience. A pipeline for analyzing the connections between a collection of documents. A pipeline for analyzing the connections between a collection of documents. A two-stage method for efficiently computing cross-document links between claims that support or contradict each other, enabling “reference-free” fact checking. A two-stage method for efficiently computing cross-document links between claims that support or contradict each other, enabling “reference-free” fact checking. A framework for visualizing cross-document connections, and integrating claims from multiple documents into a single reading experience. A framework for visualizing cross-document connections, and integrating claims from multiple documents into a single reading experience. We conclude by discussing the generality and potential social benefits of NEWSSENSE. [1] https://www.allsides.com/media-bias/media-bias-ratingmethods [2] https://adfontesmedia.com/interactive-media-bias-chart/ [3] https://news.google.com/stories/ [4] https://ground.news

From Nuisance to News Sense: Abstract and Introduction

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Jeremiah Milbauer, Carnegie Mellon University, Pittsburgh PA, USA (email: {jmilbaue | sherryw}@cs.cmu.edu);
(2) Ziqi Ding, Carnegie Mellon University, Pittsburgh PA, USA (e-mail: {ziqiding | zhijinw}@andrew.cmu.edu)
(3) Tongshuang Wu, Carnegie Mellon University, Pittsburgh PA, USA. Table of Links Abstract and Introduction
Related Work
The NEWSSENSE Framework
Pilot Study
System Overview
Discussion
Conclusion, Limitations and Ethics, and References
A. Appendix: Prompt for Claim Extraction 5 System Overview Following our user study, we implemented the NEWSSENSE framework as a browser plugin, which adds augmentations to news articles encountered on the web. Figure 1 shows the final appearance of the browser plugin. Code for system and plugin can be accessed at github.com/jmilbauer/NewsSense, and a demo video can be viewed at youtu.be/2D5LYbsQJak. This section provides a description of the natural language processing system which powers NEWSSENSE. Figure 2 illustrates the four general steps of the pipeline: Collection, Selection, Filtering, and Linking. 5.1 Article Collection First, we must collect a cluster of news articles that are all about the same news event. Our implementation scrapes data from Google News Stories, a website that collects many articles about the same events across news venues. After collecting article URLs via Google News Stories, we then collect the content of each article. A typical story contains over 50 articles. 5.2 Claim Selection The next phase of the pipeline is to select the claims within each article cluster. We initially assumed a 1-to-1 mapping between sentences and claims, but quickly found that news articles often contain complex multi-clause sentences, which are not suitable for natural language inference. To address this issue, we few-shot prompting to generate a list of claims from sentences using a large language model (LLM). In our experiments, prompt exemplars are drawn from the PROPSEGMENT dataset (Chen et al., 2022), and the LLM used is OpenAI text-davinci-003. Full prompt details are provided in Appendix A. We also note that the authors of PROPSEGMENT report that T5-Large performs reasonably well on the task, suggesting the possibility for further pipeline improvements. 5.3 Claim Filtering Articles often contain over 30 sentences. For a cluster of 50 articles, a pairwise comparison of the full cartesian product of sentences has O((NL) 2 ), which is in practice well over 1,000,000 comparisons. Performing this level of computation at scale, even if we are pre-computing results for each article cluster, is simply not feasible. To address this, we perform an initial filtering step with leverages the fact that the vast majority of claims across any two articles are unrelated. We consider two approaches for claim filtering: Embedding Similarity filtering (ES) and Lexical Overlap filtering (LeO). For Embedding Similarity filtering, we encode each claim in each article using a Transformerbased sentence encoder. Then, for each claim we retain only the k most similar other claims for comparison. In our implementation, we use the Sentence Transformers (Reimers and Gurevych, 2019) all-Mini-LM-L6-v2. For Lexical Overlap filtering, we compare each sentence only with sentences that have overlapping words, as these sentences are likely to discuss similar topics. In our implementation, we process claims by first remove stopwords, then stemming using the NLTK (Loper and Bird, 2002) implementation of the Porter Stemmer (Porter, 1980), and compute overlap scores using the Jaccard Index. We evaluated each filtering method on the MNLI (Williams et al., 2017) validation data, treating pairs of randomly sampled sentences as negative examples, and labeled “entailment" and “contradiction" sentence pairs as positive examples. For ES, we set a threshold of 0.3 cosine similarity; for LeO, we set a threshold of 0.1 overlap. We note that all-Mini-LM-L6-v2 included MNLI in its training data. We include a summary of the results of these experiments in Table 1, which indicates that the ES method outperforms the LeO method. Of particular interest is the true negative rate, as this indicates the percentage of non-related sentences we expect to filter out. 5.4 Claim Linking Once claim pairs have been filtered, we classify each pair according to the Natural Language Inference (NLI) framework, as “entailment," “contradiction," or “neutral." We employ a pretrained language model, RoBERTa (Liu et al., 2019), which was then fine-tuned on MNLI (Williams et al., 2017), a popular dataset for NLI. We download this fine-tuned version of RoBERTa from the Hugging Face model library [11]. To avoid clutter, we keep fewer than 100 of the most confident predictions for each positive class (entailment or contradiction) within the article cluster. Claims are then assigned back to the sentences from which they were generated, and the sentence pairs are linked. [11] https://huggingface.co/roberta-large-mnli This paper is available on arxiv under CC 4.0 license. Authors: (1) Jeremiah Milbauer, Carnegie Mellon University, Pittsburgh PA, USA (email: {jmilbaue | sherryw}@cs.cmu.edu); (2) Ziqi Ding, Carnegie Mellon University, Pittsburgh PA, USA (e-mail: {ziqiding | zhijinw}@andrew.cmu.edu) (3) Tongshuang Wu, Carnegie Mellon University, Pittsburgh PA, USA. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Jeremiah Milbauer, Carnegie Mellon University, Pittsburgh PA, USA (email: {jmilbaue | sherryw}@cs.cmu.edu); (2) Ziqi Ding, Carnegie Mellon University, Pittsburgh PA, USA (e-mail: {ziqiding | zhijinw}@andrew.cmu.edu) (3) Tongshuang Wu, Carnegie Mellon University, Pittsburgh PA, USA. Table of Links Abstract and Introduction Related Work The NEWSSENSE Framework Pilot Study System Overview Discussion Conclusion, Limitations and Ethics, and References A. Appendix: Prompt for Claim Extraction Abstract and Introduction Abstract and Introduction Related Work Related Work The NEWSSENSE Framework The NEWSSENSE Framework Pilot Study Pilot Study System Overview System Overview Discussion Discussion Conclusion, Limitations and Ethics, and References Conclusion, Limitations and Ethics, and References A. Appendix: Prompt for Claim Extraction A. Appendix: Prompt for Claim Extraction 5 System Overview Following our user study, we implemented the NEWSSENSE framework as a browser plugin, which adds augmentations to news articles encountered on the web. Figure 1 shows the final appearance of the browser plugin. Code for system and plugin can be accessed at github.com/jmilbauer/NewsSense, and a demo video can be viewed at youtu.be/2D5LYbsQJak. This section provides a description of the natural language processing system which powers NEWSSENSE. Figure 2 illustrates the four general steps of the pipeline: Collection, Selection, Filtering, and Linking. 5.1 Article Collection First, we must collect a cluster of news articles that are all about the same news event. Our implementation scrapes data from Google News Stories, a website that collects many articles about the same events across news venues. After collecting article URLs via Google News Stories, we then collect the content of each article. A typical story contains over 50 articles. 5.2 Claim Selection The next phase of the pipeline is to select the claims within each article cluster. We initially assumed a 1-to-1 mapping between sentences and claims, but quickly found that news articles often contain complex multi-clause sentences, which are not suitable for natural language inference. To address this issue, we few-shot prompting to generate a list of claims from sentences using a large language model (LLM). In our experiments, prompt exemplars are drawn from the PROPSEGMENT dataset (Chen et al., 2022), and the LLM used is OpenAI text-davinci-003. Full prompt details are provided in Appendix A. We also note that the authors of PROPSEGMENT report that T5-Large performs reasonably well on the task, suggesting the possibility for further pipeline improvements. 5.3 Claim Filtering Articles often contain over 30 sentences. For a cluster of 50 articles, a pairwise comparison of the full cartesian product of sentences has O((NL) 2 ), which is in practice well over 1,000,000 comparisons. Performing this level of computation at scale, even if we are pre-computing results for each article cluster, is simply not feasible. To address this, we perform an initial filtering step with leverages the fact that the vast majority of claims across any two articles are unrelated. We consider two approaches for claim filtering: Embedding Similarity filtering (ES) and Lexical Overlap filtering (LeO). For Embedding Similarity filtering, we encode each claim in each article using a Transformerbased sentence encoder. Then, for each claim we retain only the k most similar other claims for comparison. In our implementation, we use the Sentence Transformers (Reimers and Gurevych, 2019) all-Mini-LM-L6-v2. k For Lexical Overlap filtering, we compare each sentence only with sentences that have overlapping words, as these sentences are likely to discuss similar topics. In our implementation, we process claims by first remove stopwords, then stemming using the NLTK (Loper and Bird, 2002) implementation of the Porter Stemmer (Porter, 1980), and compute overlap scores using the Jaccard Index. We evaluated each filtering method on the MNLI (Williams et al., 2017) validation data, treating pairs of randomly sampled sentences as negative examples, and labeled “entailment" and “contradiction" sentence pairs as positive examples. For ES, we set a threshold of 0.3 cosine similarity; for LeO, we set a threshold of 0.1 overlap. We note that all-Mini-LM-L6-v2 included MNLI in its training data. We include a summary of the results of these experiments in Table 1, which indicates that the ES method outperforms the LeO method. Of particular interest is the true negative rate, as this indicates the percentage of non-related sentences we expect to filter out. 5.4 Claim Linking Once claim pairs have been filtered, we classify each pair according to the Natural Language Inference (NLI) framework, as “entailment," “contradiction," or “neutral." We employ a pretrained language model, RoBERTa (Liu et al., 2019), which was then fine-tuned on MNLI (Williams et al., 2017), a popular dataset for NLI. We download this fine-tuned version of RoBERTa from the Hugging Face model library [11]. To avoid clutter, we keep fewer than 100 of the most confident predictions for each positive class (entailment or contradiction) within the article cluster. Claims are then assigned back to the sentences from which they were generated, and the sentence pairs are linked. [11] https://huggingface.co/roberta-large-mnli

From Nuisance to News Sense: System Overview

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Jeremiah Milbauer, Carnegie Mellon University, Pittsburgh PA, USA (email: {jmilbaue | sherryw}@cs.cmu.edu);
(2) Ziqi Ding, Carnegie Mellon University, Pittsburgh PA, USA (e-mail: {ziqiding | zhijinw}@andrew.cmu.edu)
(3) Tongshuang Wu, Carnegie Mellon University, Pittsburgh PA, USA. Table of Links Abstract and Introduction
Related Work
The NEWSSENSE Framework
Pilot Study
System Overview
Discussion
Conclusion, Limitations and Ethics, and References
A. Appendix: Prompt for Claim Extraction 4 Pilot Study To gather feedback on the proposed NEWSSENSE interface and provide insights for the actual implementation, we conducted a pilot user study using a NewsReader mockup built with Figma [10]. This section describes the design and results of the study. 4.1 Study Design We aim to collect feedback on NEWSSENSE’s basic functionality, interface design, and content quality. The participants were assigned a task of reading a news article using NEWSSENSE and answering a set of questions. The questions focused on the content of the news, how and where the user located information, and their level of trust in the information. These questions aimed to assess the basic functionality of NEWSSENSE in helping readers understand news comprehensively, to motivate further development of the system. 4.2 Results Following the pilot user study with over 10 users, we identified several key findings. First, all users found NEWSSENSE to be useful in locating important information and verifying the credibility of news articles, aligning with our initial goal. The user-friendly interface of NEWSSENSE was wellreceived, though participants suggested enhancing interactivity to set it apart from other solutions. For instance, displaying real-time feedback like “NewsSense is analyzing the article" during loading. Regarding content quality, some users found NEWSSENSE limited and suggested increased labeling or categorization within articles. One user noted Two highlighted sentences per page are insufficient for in-depth analysis." User preferences varied for article summarization, with some wanting more key points and others preferring brevity. Contradicting previous feedback, one user preferred “Summarizing key points only, rather than selecting sentences with unclear relevance." Addressing this, NEWSSENSE could allow customization, letting users choose key point count and filter supporting/- contradicting data. 4.3 Study Takeaways We found that users liked how NewsSense highlighted important sentences from an article. We realized that the claims which are consistent across multiple articles (ie, those which are supported at least once) are likely to be the most important aspects to a given story. NEWSSENSE could inform readers when there are key claims from across the article cluster missing from the article they are reading. We also found that the bias labels for news venues could be overwhelming, and including them ran counter to our aim of reference-free verification; we eliminated these labels. Users also appreciated how highlighted sentences functioned as summaries. Consequently we enhance the visibility of text highlights and further emphasize the alignment or contradiction of specific source by making the External Evidence cards colored accordingly. [10] https://www.figma.com/ This paper is available on arxiv under CC 4.0 license. Authors: (1) Jeremiah Milbauer, Carnegie Mellon University, Pittsburgh PA, USA (email: {jmilbaue | sherryw}@cs.cmu.edu); (2) Ziqi Ding, Carnegie Mellon University, Pittsburgh PA, USA (e-mail: {ziqiding | zhijinw}@andrew.cmu.edu) (3) Tongshuang Wu, Carnegie Mellon University, Pittsburgh PA, USA. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Jeremiah Milbauer, Carnegie Mellon University, Pittsburgh PA, USA (email: {jmilbaue | sherryw}@cs.cmu.edu); (2) Ziqi Ding, Carnegie Mellon University, Pittsburgh PA, USA (e-mail: {ziqiding | zhijinw}@andrew.cmu.edu) (3) Tongshuang Wu, Carnegie Mellon University, Pittsburgh PA, USA. Table of Links Abstract and Introduction Related Work The NEWSSENSE Framework Pilot Study System Overview Discussion Conclusion, Limitations and Ethics, and References A. Appendix: Prompt for Claim Extraction Abstract and Introduction Abstract and Introduction Related Work Related Work The NEWSSENSE Framework The NEWSSENSE Framework Pilot Study Pilot Study System Overview System Overview Discussion Discussion Conclusion, Limitations and Ethics, and References Conclusion, Limitations and Ethics, and References A. Appendix: Prompt for Claim Extraction A. Appendix: Prompt for Claim Extraction 4 Pilot Study To gather feedback on the proposed NEWSSENSE interface and provide insights for the actual implementation, we conducted a pilot user study using a NewsReader mockup built with Figma [10]. This section describes the design and results of the study. 4.1 Study Design We aim to collect feedback on NEWSSENSE’s basic functionality, interface design, and content quality. The participants were assigned a task of reading a news article using NEWSSENSE and answering a set of questions. The questions focused on the content of the news, how and where the user located information, and their level of trust in the information. These questions aimed to assess the basic functionality of NEWSSENSE in helping readers understand news comprehensively, to motivate further development of the system. 4.2 Results Following the pilot user study with over 10 users, we identified several key findings. First, all users found NEWSSENSE to be useful in locating important information and verifying the credibility of news articles, aligning with our initial goal. The user-friendly interface of NEWSSENSE was wellreceived, though participants suggested enhancing interactivity to set it apart from other solutions. For instance, displaying real-time feedback like “NewsSense is analyzing the article" during loading. “NewsSense is analyzing the article" Regarding content quality, some users found NEWSSENSE limited and suggested increased labeling or categorization within articles. One user noted Two highlighted sentences per page are insufficient for in-depth analysis ." User preferences varied for article summarization, with some wanting more key points and others preferring brevity. Contradicting previous feedback, one user preferred “ Summarizing key points only, rather than selecting sentences with unclear relevance ." Addressing this, NEWSSENSE could allow customization, letting users choose key point count and filter supporting/- contradicting data. Two highlighted sentences per page are insufficient for in-depth analysis Summarizing key points only, rather than selecting sentences with unclear relevance 4.3 Study Takeaways We found that users liked how NewsSense highlighted important sentences from an article. We realized that the claims which are consistent across multiple articles (ie, those which are supported at least once) are likely to be the most important aspects to a given story. NEWSSENSE could inform readers when there are key claims from across the article cluster missing from the article they are reading. NewsSense We also found that the bias labels for news venues could be overwhelming, and including them ran counter to our aim of reference-free verification; we eliminated these labels. Users also appreciated how highlighted sentences functioned as summaries. Consequently we enhance the visibility of text highlights and further emphasize the alignment or contradiction of specific source by making the External Evidence cards colored accordingly. [10] https://www.figma.com/

From Nuisance to News Sense: Pilot Study

Stories, News & Mentions about SENSE

Sense’s Stories on HackerNoon