Clearly

clearly

How to Write Clearly: Rules and Exercises on English Composition by Edwin Abbott Abbott, is part of the HackerNoon Books Series. You can jump to any chapter in this book . PREFACE. here PREFACE. Almost every English boy can be taught to write clearly, so far at least as clearness depends upon the arrangement of words. Force, elegance, and variety of style are more difficult to teach, and far more difficult to learn; but clear writing can be reduced to rules. To teach the art of writing clearly is the main object of these Rules and Exercises. Ambiguity may arise, not only from bad arrangement, but also from other causes—from the misuse of single words, and from confused thought. These causes are not removable by definite rules, and therefore, though not neglected, are not prominently considered in this book. My object rather is to point out some few continually recurring causes of ambiguity, and to suggest definite remedies in each case. Speeches in Parliament, newspaper narratives and articles, and, above all, resolutions at public meetings, furnish abundant instances of obscurity arising from the monotonous neglect of some dozen simple rules. The art of writing forcibly is, of course, a valuable acquisition—almost as valuable as the art of writing clearly. But forcible expression is not, like clear expression, a mere question of mechanism and of the manipulation of words; it is a much higher power, and implies much more. Writing clearly does not imply thinking clearly. A man may think and reason as obscurely as Dogberry himself, but he may (though it is not probable that he will) be able to write clearly for all that. Writing clearly—so far as arrangement of words is concerned—is a mere matter of adverbs, conjunctions, prepositions, and auxiliary verbs, placed and repeated according to definite rules.[1] Even obscure or illogical thought can be clearly expressed; indeed, the transparent medium of clear writing is not least beneficial when it reveals the illogical nature of the meaning beneath it. On the other hand, if a man is to write forcibly, he must (to use a well-known illustration) describe Jerusalem as "sown with salt," not as "captured," and the Jews not as being "subdued" but as "almost exterminated" by Titus. But what does this imply? It implies knowledge, and very often a great deal of knowledge, and it implies also a vivid imagination. The writer must have eyes to see the vivid side of everything, as well as words to describe what he sees. Hence forcible writing, and of course tasteful writing also, is far less a matter of rules than is clear writing; and hence, though forcible writing is exemplified in the exercises, clear writing occupies most of the space devoted to the rules. Boys who are studying Latin and Greek stand in especial need of help to enable them to write a long English sentence clearly. The periods of Thucydides and Cicero are not easily rendered into our idiom without some knowledge of the links that connect an English sentence. There is scarcely any better training, rhetorical as well as logical, than the task of construing Thucydides into genuine English; but the flat, vague, long-winded Greek-English and Latin-English imposture that is often tolerated in our examinations and is allowed to pass current for genuine English, diminishes instead of increasing the power that our pupils should possess over their native language. By getting marks at school and college for construing good Greek and Latin into bad English, our pupils systematically unlearn what they may have been allowed to pick up from Milton and from Shakespeare. I must acknowledge very large obligations to Professor Bain's treatise on "English Composition and Rhetoric," and also to his English Grammar. I have not always been able to agree with Professor Bain as to matters of taste; but I find it difficult to express my admiration for the systematic thoroughness and suggestiveness of his book on Composition. In particular, Professor Bain's rule on the use of "that" and "which" (see Rule 8) deserves to be better known.[2] The ambiguity produced by the confusion between these two forms of the Relative is not a mere fiction of pedants; it is practically serious. Take, for instance, the following sentence, which appeared lately in one of our ablest weekly periodicals: "There are a good many Radical members in the House cannot forgive the Prime Minister for being a Christian." Twenty years hence, who is to say whether the meaning is " , i.e. members in the House," or "there are a good many Radical members of the House cannot &c."? Professor Bain, apparently admitting no exceptions to his useful rule, amends many sentences in a manner that seems to me intolerably harsh. Therefore, while laying due stress on the utility of the rule, I have endeavoured to point out and explain the exceptions. who and they all the Radical that The rules are stated as briefly as possible, and are intended not so much for use by themselves as for reference while the pupil is working at the exercises. Consequently, there is no attempt to prove the rules by accumulations of examples. The few examples that are given, are given not to prove, but to illustrate the rules. The exercises are intended to be written out and revised, as exercises usually are; but they may also be used for instruction. The books being shut, the pupils, with their written exercises before them, may be questioned as to the reasons for the several alterations they have made. Experienced teachers will not require any explanation of the arrangement or rather non-arrangement of the exercises. They have been purposely mixed together unclassified to prevent the pupil from relying upon anything but his own common sense and industry, to show him what is the fault in each case, and how it is to be amended. Besides references to the rules, notes are attached to each sentence, so that the exercises ought not to present any difficulty to a painstaking boy of twelve or thirteen, provided he has first been fairly trained in English grammar. vivâ voce The "Continuous Extracts" present rather more difficulty, and are intended for boys somewhat older than those for whom the Exercises are intended. The attempt to modernize, and clarify, so to speak, the style of Burnet, Clarendon, and Bishop Butler,[3] may appear ambitious, and perhaps requires some explanation. My object has, of course, not been to the style of these authors, but to show how their meaning might be expressed more clearly in modern English. The charm of the style is necessarily lost, but if the loss is recognized both by teacher and pupil, there is nothing, in my opinion, to counterbalance the obvious utility of such exercises. Professor Bain speaks to the same effect:[4] "For an English exercise, the matter should in some way or other be supplied, and the pupil disciplined in giving it expression. I know of no better method than to prescribe passages containing good matter, but in some respects imperfectly worded, to be amended according to the laws and the proprieties of style. Our older writers might be extensively, though not exclusively, drawn upon for this purpose." improve upon To some of the friends whose help has been already acknowledged in "English Lessons for English People," I am indebted for further help in revising these pages. I desire to express especial obligations to the Rev. J. H. Lupton, late Fellow of St. John's College, Cambridge, and Second Master of St. Paul's School, for copious and valuable suggestions; also to several of my colleagues at the City of London School, among whom I must mention in particular the Rev. A. R. Vardy, Fellow of Trinity College, Cambridge. * * * * * Before electrotyping the Fourth and Revised Edition, I wish to say one word as to the manner in which this book has been used by my highest class, as a collection of Rules for reference in their construing lessons. In construing, from Thucydides especially, I have found Rules 5, 30, 34, 36, 37, and 40_a_, of great use. The rules about Metaphor and Climax have also been useful in correcting faults of taste in their Latin and Greek compositions. I have hopes that, used in this way, this little book may be of service to the highest as well as to the middle classes of our schools. FOOTNOTES: [1] Punctuation is fully discussed in most English Grammars, and is therefore referred to in this book only so far as is necessary to point out the slovenly fault of trusting too much to punctuation, and too little to arrangement. [2] Before meeting with Professor Bain's rule, I had shown that the difference between the Relatives is generally observed the Shakespeare. See "Shakespearian Grammar," paragraph 259. [3] Sir Archibald Alison stands on a very different footing. The extracts from this author are intended to exhibit the dangers of verbosity and exaggeration. [4] "English Composition and Rhetoric," p. vii. About HackerNoon Book Series: We bring you the most important technical, scientific, and insightful public domain books. This book is part of the public domain. Edwin Abbott Abbott (2007). How to Write Clearly: Rules and Exercises on English Composition. Urbana, Illinois: Project Gutenberg. Retrieved https://www.gutenberg.org/cache/epub/22600/pg22600.html This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at , located at . www.gutenberg.org https://www.gutenberg.org/policy/license.html

Almost every English boy can be taught to write clearly

How to Write Clearly: Rules and Exercises on English Composition by Edwin Abbott Abbott is part of the HackerNoon Books Series. Read this book online for free on HackerNoon! : How to Write Clearly: Rules and Exercises on English Composition Title : Edwin Abbott Abbott Author : September 14, 2007 [eBook #22600] Most recently updated: January 2, 2021 Release date : English Language TABLE OF LINKS PREFACE INDEX OF RULES. I. CLEARNESS AND FORCE. . WORDS ORDER OF WORDS IN A SENTENCE. II. BREVITY. CLEARNESS AND FORCE. WORDS. ORDER OF WORDS IN A SENTENCE. BREVITY. EXERCISES CONTINUOUS EXERCISES CLEARNESS. BURNET. BISHOP BUTLER. BREVITY. About HackerNoon Book Series: We bring you the most important technical, scientific, and insightful public domain books. This book is part of the public domain. Edwin Abbott Abbott (2007). How to Write Clearly: Rules and Exercises on English Composition. Urbana, Illinois: Project Gutenberg. Retrieved https://www.gutenberg.org/cache/epub/22600/pg22600.html This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at , located at . www.gutenberg.org https://www.gutenberg.org/policy/license.html

How to Write Clearly: Rules and Exercises on English Composition - Table of Links

Dependency injection (DI) is a wonderful thing. Simply add your dependency as a parameter to the constructor (most commonly) of your class, register it with you DI container, and away you go - the DI container will manage the rest. Some of the key benefits of DI are: greater testability, greater maintainability, and greater reusability. : { { ( dbContent = MyDbContext())
        {
            dbContext.Orders.Add(order);
            dbContent.SaveChanges(); Ok();
        }
    } { ( dbContext = MyDbContext())
        { orders = dbContext.Orders.ToList(); JsonResult(orders);
        }
    }
} : { MyDbContext _dbContext; {
        _dbContext = dbContext;
    } {
        _dbContext.Orders.Add(order);
        _dbContent.SaveChanges(); Ok();
    } { orders = _dbContext.Orders.ToList(); JsonResult(orders);
    }
} // without DI public class OrderController Controller ActionResult ( ) public Post Order order using var new return ActionResult ( ) public Get using var new var return new // with DI public class OrderController Controller private readonly ( ) public OrderController MyDbContext dbContext ActionResult ( ) public Post Order order return ActionResult ( ) public Get var return new However, I recently came across a use case where DI can be a real pain - dependency injection in inherited classes. The Problem I have recently been working a lot with the package, using it's request/request handler pattern to issue commands in the system (inspired by ). Mediatr Jason Taylor's Clean Architecture solution I typically create a RequestHandler base class that contains common dependencies and functionality. Each concrete request handler can then inherit from this base class. < , > : < , > : < >
{ {
        DbContext = dbContext;
    } IApplicationDbContext DbContext { ; } ;
} MyRequestHandler : RequestHandler<MyRequest, MyResponse>
{ { } { }
} public abstract class RequestHandler TRequest TResponse IRequestHandler TRequest TResponse where TRequest IRequest TResponse ( ) public RequestHandler IApplicationDbContext dbContext protected get Task<TResponse> ( ) public abstract Handle TRequest request, CancellationToken cancellationToken public ( ) : ( ) public MyRequestHandler IApplicationDbContext dbContext base dbContext Task<MyResponse> ( ) public override Handle MyRequest request, CancellationToken cancellationToken // handler logic The problem comes when I want to add more dependencies to the base class. Now I have to go through to every single concrete request handler and update the constructor to take the new dependency as well. Fortunately, the code will not compile if I miss one, so there is no risk of a runtime error, but it is still incredibly tedious work to have to update every single request handler. Also, you can end up with very large constructors, which obscures the intention of the class. < , > : < , > : < >
{ {
        DbContext = dbContext;
        CurrentUser = currentUser;
    } IApplicationDbContext DbContext { ; } ICurrentUser CurrentUser { ; } ;
} MyRequestHandler : RequestHandler<MyRequest, MyResponse>
{ { } { }
} public abstract class RequestHandler TRequest TResponse IRequestHandler TRequest TResponse where TRequest IRequest TResponse ( ) public RequestHandler IApplicationDbContext dbContext, ICurrentUser currentUser protected get protected get Task<TResponse> ( ) public abstract Handle TRequest request, CancellationToken cancellationToken public ( ) : ( ) public MyRequestHandler IApplicationDbContext dbContext, ICurrentUser currentUser base dbContext, currentUser Task<MyResponse> ( ) public override Handle MyRequest request, CancellationToken cancellationToken // handler logic The Solution - Dependency Aggregates The solution to this problem is really quite simple. Rather than injecting the dependencies directly, create a new class that contains the dependencies (known as an aggregate) and inject that instead. {
    IApplicationDbContext DbContext { ; }
    ICurrentUser { ; }
} : { {
        DbContext = dbContext;
        CurrentUser = currentUser;
    } IApplicationDbContext DbContext { ; } ICurrentUser CurrentUser { ; }
} < , > : < , > : < >
{ {
        DbContext = aggregate.DbContext;
        CurrentUser = aggregate.CurrentUser;
    } IApplicationDbContext DbContext { ; } ICurrentUser CurrentUser { ; } ;
} MyRequestHandler : RequestHandler<MyRequest, MyResponse>
{ { } { }
} public interface IDependencyAggregate get get public class DependencyAggregate IDependencyAggregate ( ) public DependencyAggregate IApplicationDbContext dbContext, ICurrentUser currentUser public get public get public abstract class RequestHandler TRequest TResponse IRequestHandler TRequest TResponse where TRequest IRequest TResponse ( ) public RequestHandler IDependencyAggregate aggregate protected get protected get Task<TResponse> ( ) public abstract Handle TRequest request, CancellationToken cancellationToken public ( ) : ( ) public MyRequestHandler IDependencyAggregate aggregate base aggregate Task<MyResponse> ( ) public override Handle MyRequest request, CancellationToken cancellationToken // handler logic Now if I want to add a new dependency, the only places that I need to change the code are in the DependencyAggregate class and the RequestHandler base class (I don't need to make any changes to the inherited classes). Conclusion In this post I have described a simple method for managing dependency injection in inherited classes, by creating a dependency aggregate class to inject into the base class. This ensures that new dependencies can easily be introduced with having to make changes to every inherited class. I post mostly about full stack .NET and Vue web development. To make sure that you don't miss out on any posts, please follow this blog and . If you found this post helpful, please like it and share it. You can also find me on . subscribe to my newsletter Twitter Previously published at https://samwalpole.com/handling-dependency-injection-in-inherited-classes

How to Create a Dependency Aggregate Class to Manage DI: Clearly Exemplified

I lead the technology industry business unit for a fast-growing analytics firm. And one of the perks of my job is the numerous conversations I get to have with a variety of prospects or clients. I recently read the novel ‘ by Rolf Dobelli, which deals with systematic cognitive errors of human beings as a result of our evolution. The Art of Thinking Clearly’ I enjoyed reading the book immensely, relating those cognitive errors to what I experience every day. I am selecting a few rampant cognitive errors mentioned in the book and linking them to situations I have come across. Neo-mania Def: The mania for all things new and shiny. The desire to have automation, machine models, artificial intelligence, and other such cutting-edge capabilities is intense in organizations today and probably justified. However, it’s alarming to see many of these efforts promise a very large ROI but never deliver it. marketing learning I once had an executive of a large organization say without batting an eyelid - “We have invested millions in a big data environment, and we now need to figure how to use it.” I am always disappointed when I hear things like — “Can you tell me how I can use machine learning capabilities in my business/function?” I wonder if anyone would say it if I replaced ‘machine learning’ with ‘statistics’. Conversely, the best use of ML/AI is when the need or a business problem was identified first and ML/AI emerged as a solution. Swimmer’s Body Illusion Def: Confusing the process of selection for effect. Rolf explains in the book that swimmers don’t have great bodies because they swim a lot; they are swimmers because their physique is suitable for swimming. I have seen companies falling to this illusion when they look to replicate an analytics success story from companies that seem to be doing very well and claiming that analytics is driving their success. It’s safe to say that the retail industry has the most mature analytics capability, but nobody is blaming the analytics team at Sears and Macy’s for their decline. In my opinion, most companies would succeed in their analytics initiatives if they just are more rational and use data to be just that. (Read more: Building an analytics-first organization ) Authority Bias Def: The tendency of human beings to follow authority without question, a culture most visible in religion but equally prominent in businesses. When I ask my clients why they were following a particular strategy, the responses most often indicate an authority bias. Interestingly, the bias seems more obvious when the perceived results from the strategy are less obvious. My advice to my clients has always been to focus on outcomes and not on the process. I am amazed even now that such a conversation isn’t always easy. The Paradox of Choice Def: Too many choices causing decision paralysis and subsequently leading to poor decisions. The number of , vendors, and technologies today is confounding. One would expect then that most clients would be happy with the choices they make. However, I haven’t seen a company that is entirely happy with its selection of business intelligence (BI) or . analytics solutions analytics platforms If only they hadn’t compared the tools with each other but against a prioritized list of their own needs, they would have come to a better decision. Default Effect/Status Quo Bias Def: A strong tendency to cling to the way things are, even when they put us at a disadvantage. Most tech companies with revenues of $500 million to $2 billion are hyper-focused on growth. The most important part of achieving growth is to size the market right, find the account segments to focus on, and then have the sales force or channel partners gain market share. One would imagine that these companies would have nailed this process well. I am yet to come across one. (Read more: Business Transformation — How Big Data Analytics Helps ) Alternative Blindness Def: Systematically forgetting to compare an existing offer with the next-best alternative. I have run across companies with no analytics capabilities looking to hire a team first, spending a million dollars, and losing quite a bit of time before they see any capability in action. When we provide them the alternative to spend less than half a million and have the capabilities stand up in six months, one would imagine it would be a no-brainer. It is, but only to our clients. *Please note that the terms and their descriptions are from Rolf Dobelli’s book. (This article was authored by Maruti Peri and first appeared on LinkedIn , the BRIDGEi2i blog & Datafloq .)

Master “The Art of Thinking Clearly” Before Making Your Analytics Investments!

In my previous installment (READ HERE) , I explored how embedding models struggle with basic language variations like unit of measurements and domain specific sentences. The community response was remarkable – clearly, many of you have encountered similar challenges in your work. Today, I'm expanding our investigation to uncover even more concerning blind spots I've identified through extensive testing. These fundamental issues have profound implications for how we approach AI system development. READ HERE) READ HERE This article is the third entry in my Hallucinations by Design series, building directly on our examination of embedding hallucinations. For optimal comprehension, I strongly suggest reading the previous articles first (HERE and HERE)  to establish the essential background needed to fully appreciate the concepts we'll be discussing here. This sequential approach will provide you with a more coherent understanding of these critical issues. (HERE and HERE) HERE HERE Statistical significance gets completely reversed My statistician colleague turned pale when I showed him this. The model rated "The results showed a significant difference (p<0.05)" and "The results showed no significant difference (p>0.05)" at 0.94 similarity. He just kept shaking his head. "That's... those are opposites. That's the whole point of statistical testing." "The results showed a significant difference (p<0.05)" and "The results showed no significant difference (p>0.05)" at 0.94 similarity. The embeddings couldn't distinguish between statistically significant and insignificant findings. Researchers searching for proven effects were shown a mix of validated and invalidated studies. Do you think scientists making research decisions appreciate confusing evidence with non-evidence? I am sure I wouldn't want my research funding wasted pursuing effects that studies actually disproved. The embeddings couldn't distinguish between statistically significant and insignificant findings. Statistical significance is the cornerstone of empirical research. When your model can't distinguish between "proven effect" and "no proven effect," you've undermined the entire scientific method. We're basically working with models that ignore p-values despite analyzing text where these values determine whether a finding is considered valid. Identity versus similarity gets muddled My materials science colleague laughed out loud when I showed her this one. The model gave a 0.97 similarity score to "This material is aluminum" versus "This material resembles aluminum." She stopped laughing when I explained how this was affecting her lab's search results. Being something and looking like something are fundamentally different! The model gave a 0.97 similarity score to "This material is aluminum" versus "This material resembles aluminum." The embeddings couldn't distinguish between materials that actually were a substance and those that merely looked or behaved similarly. Engineers searching for aluminum components were getting results for aluminum-like alloys with completely different properties. Do you think aerospace engineers appreciate getting information about aluminum-like materials when structural integrity depends on actual aluminum? I am sure I wouldn't want to fly in a plane built with "sort of like aluminum" parts. Engineers searching for aluminum components were getting results for aluminum-like alloys with completely different properties. The distinction between identity and similarity is fundamental to precise communication. When your model treats "X is Y" as equivalent to "X resembles Y," you've lost the ability to make definitive identifications. We're basically working with models that blur categorical boundaries despite analyzing text where precise identification is often the entire point. When your model treats "X is Y" as equivalent to "X resembles Y," you've lost the ability to make definitive identifications. Presuppositions vanish into thin air Now a bit from philosophy subject and this one also bothered me on a deep level. The embedding model rated "What caused the system to fail?" and "Did the system fail?" at a 0.93 similarity. That completely misses the point! The first question assumes failure happened, while the second asks if it happened at all. That's Logic 101! The embedding model rated "What caused the system to fail?" and "Did the system fail?" at a 0.93 similarity. The embeddings couldn't distinguish between questions presupposing a condition and questions asking if the condition existed. Support agents investigating system failures were getting mixed with cases questioning whether failures had occurred at all. Do you think IT managers trying to resolve incidents appreciate wasting time on cases where nothing actually broke? I am sure I wouldn't want my support team chasing phantoms instead of fixing real problems. Presuppositions fundamentally change what a sentence means. When your model treats "Why is X true?" as equivalent to "Is X true?", you've lost the ability to understand what is being assumed versus questioned. We're basically working with models that miss embedded assumptions despite analyzing text where these assumptions often contain critical information. hy is X true?" as equivalent to "Is X true?", you've lost the ability to understand what is being assumed versus questioned. Percentages get completely inverted Looking at pharmaceutical data gave me a headache that no embedding model could properly classify. Get this: "Only 5% of patients experienced side effects" versus "Up to 95% of patients experienced side effects" scored a 0.90 similarity. In what universe are those remotely the same? One's a remarkably safe drug, the other would probably never get FDA approval! Get this: "Only 5% of patients experienced side effects" versus "Up to 95% of patients experienced side effects" scored a 0.90 similarity. I discovered this building a pharmaceutical research database. The algorithm couldn't distinguish between dramatically different safety profiles. Researchers looking for treatments with minimal side effects were shown options with nearly universal adverse effects. Do you think doctors prescribing medications appreciate confusing remarkably safe drugs with overwhelmingly problematic ones? I am sure I wouldn't want to take medication with a 95% side effect rate when I thought it was 5%. Percentages express fundamentally different magnitudes that often determine risk assessments and decision-making. When your model treats "5%" as similar to "95%," you've lost the ability to understand statistical significance. We're basically working with models that see percentages as decorative rather than substantive despite analyzing text where these values drive critical decisions. When your model treats "5%" as similar to "95%," you've lost the ability to understand statistical significance. There is a way to fix these kinds of issues and they will be covered later. Let's understand more issues with embeddings. Metaphors versus literal language - all the same! Remember when embedding models were supposed to understand context? Well, they don't. "The market is climbing a wall of worry" versus "Rock climbers are scaling a worrying wall" scored an 0.89 similarity. Anyone with basic reading comprehension knows one's a financial metaphor and one's about actual mountain climbing. Remember when embedding models were supposed to understand context? Well, they don't. "The market is climbing a wall of worry" versus "Rock climbers are scaling a worrying wall" scored an 0.89 similarity. I discovered this building a financial news analysis system. The algorithm couldn't distinguish between metaphorical and literal language. Investors searching for market analyses were getting mixed results including actual rock climbing stories. Do you think traders making investment decisions appreciate getting literal climbing articles mixed with financial analyses? I am sure I wouldn't want my retirement savings influenced by articles about rock scaling techniques. Metaphorical language is pervasive in specialized domains like finance, medicine, and law. When your model confuses metaphors with their literal interpretations, you've lost the ability to understand domain jargon. We're basically working with models that miss figurative meaning despite analyzing text filled with specialized metaphors that domain experts instantly recognize. Extensional versus intensional references get confused I'm not an astronomer, but even I know this is wrong. "The Morning Star is visible at dawn" versus "The Evening Star is visible at dawn" scored a 0.93 similarity. Here's the kicker - both refer to Venus, but only one statement is actually true! The Evening Star (Venus in the evening) isn't visible at dawn, by definition. "The Morning Star is visible at dawn" versus "The Evening Star is visible at dawn" scored a 0.93 similarity. The embeddings couldn't distinguish between different ways of referring to the same object when those references carried different truth values. Astronomers searching for accurate observation times were shown contradictory information. Do you think researchers planning observations appreciate getting objectively false viewing times? I am sure I wouldn't want to wake up at dawn to see something that's only visible at dusk. Different ways of referring to the same entity often carry different contextual implications. When your model treats all references to an entity as interchangeable, you've lost the ability to preserve context-dependent truth. We're basically working with models that collapse referential distinctions despite analyzing text where these distinctions determine factual accuracy. Domain-specific thresholds get completely missed My doctor friend nearly had a heart attack when I showed her this test result. "The patient's fever was 101°F" versus "The patient's fever was 104°F" scored a 0.97 similarity. "Are you KIDDING me?" she shouted. "That's the difference between 'take some Tylenol' and 'get to the ER immediately'!" "The patient's fever was 101°F" versus "The patient's fever was 104°F" scored a 0.97 similarity. The embeddings couldn't distinguish between clinically significant temperature thresholds. Doctors searching for cases of dangerous fevers were getting mixed results including mild temperature elevations. Do you think emergency physicians appreciate getting non-urgent cases mixed with life-threatening ones? I am sure I wouldn't want my dangerously ill child triaged as having a mild fever. Domain-specific thresholds often represent critical decision boundaries. When your model treats "just above normal" the same as "critically elevated," you've lost the ability to distinguish between routine and emergency situations. We're basically working with models that see numbers as interchangeable despite analyzing text where small numerical differences represent hugely different clinical situations. Date formats cause international incidents Ever missed a deadline because of date format confusion? Our models do it consistently. "Submit your application by 12/10/2023" versus "Submit your application by 10/12/2023" scored an almost perfect 0.99 similarity. Depending on whether you're in the US or Europe, those dates are two months apart! "Submit your application by 12/10/2023" versus "Submit your application by 10/12/2023" scored an almost perfect 0.99 similarity. The embeddings couldn't distinguish between MM/DD and DD/MM formats. Students applying to universities were missing deadlines because date formats were interpreted differently across countries. Do you think applicants appreciate missing life-changing opportunities because of date format confusion? I am sure I wouldn't want my future derailed because an AI couldn't tell October from December. The embeddings couldn't distinguish between MM/DD and DD/MM formats Date format ambiguity isn't just annoying - it can have legal, financial, and personal consequences. When your model treats different date formats as identical, you've introduced a cultural bias that particularly impacts international systems. We're basically working with models that ignore date format conventions despite analyzing text where these distinctions can determine whether something is on time or hopelessly late. Date format ambiguity isn't just annoying - it can have legal, financial, and personal consequences. The Truth and the Results Here is the comparison between msmarco-distilbert-base-tas-b, all-mpnet-base-v2, and open-ai-text-embedding-3-large, and you will notice that there is no significant difference between the output of these models. msmarco-distilbert-base-tas-b embedding score across different test cases msmarco-distilbert-base-tas-b embedding score across different test cases all-mpnet-base-v2 embedding score across different test cases all-mpnet-base-v2 embedding score across different test cases openai-text-embedding-3-large embedding score across different test cases openai-text-embedding-3-large embedding score across different test cases Cannot stress more.. Look, embeddings are amazingly useful despite these problems. I'm not advocating against using them, but rather, it's crucial to approach them cautiously. Here's my battle-tested advice after dozens of projects and countless failures: Test your model on real user language patterns before deployment. Not academic benchmarks, not sanitized test cases – actual examples of how your users communicate. We built a "linguistic stress test" toolkit that simulates common variations like negations, typos, and numerical differences. Every system we test fails in some areas – the question is whether those areas matter for your specific application.


Build guardrails around critical blind spots. Different applications have different can't-fail requirements. For healthcare, it's typically negation and entity precision. For finance, it's numbers and temporal relationships. For legal, it's conditions and obligations. Identify what absolutely can't go wrong in your domain, and implement specialized safeguards.


Layer different techniques instead of betting everything on embeddings. Our most successful systems combine embedding-based retrieval with keyword verification, explicit rule checks, and specialized classifiers for critical distinctions. This redundancy isn't inefficient; it's essential.


Be transparent with users about what the system can and can't do reliably. We added confidence scores that explicitly flag when a result might involve negation, numerical comparison, or other potential weak points. Users appreciate the honesty, and it builds trust in the system overall. Test your model on real user language patterns before deployment. Not academic benchmarks, not sanitized test cases – actual examples of how your users communicate. We built a "linguistic stress test" toolkit that simulates common variations like negations, typos, and numerical differences. Every system we test fails in some areas – the question is whether those areas matter for your specific application. Test your model on real user language patterns before deployment. Not academic benchmarks, not sanitized test cases – actual examples of how your users communicate. We built a "linguistic stress test" toolkit that simulates common variations like negations, typos, and numerical differences. Every system we test fails in some areas – the question is whether those areas matter for your specific application. Build guardrails around critical blind spots. Different applications have different can't-fail requirements. For healthcare, it's typically negation and entity precision. For finance, it's numbers and temporal relationships. For legal, it's conditions and obligations. Identify what absolutely can't go wrong in your domain, and implement specialized safeguards. Build guardrails around critical blind spots. Different applications have different can't-fail requirements. For healthcare, it's typically negation and entity precision. For finance, it's numbers and temporal relationships. For legal, it's conditions and obligations. Identify what absolutely can't go wrong in your domain, and implement specialized safeguards. Layer different techniques instead of betting everything on embeddings. Our most successful systems combine embedding-based retrieval with keyword verification, explicit rule checks, and specialized classifiers for critical distinctions. This redundancy isn't inefficient; it's essential. Layer different techniques instead of betting everything on embeddings. Our most successful systems combine embedding-based retrieval with keyword verification, explicit rule checks, and specialized classifiers for critical distinctions. This redundancy isn't inefficient; it's essential. Be transparent with users about what the system can and can't do reliably. We added confidence scores that explicitly flag when a result might involve negation, numerical comparison, or other potential weak points. Users appreciate the honesty, and it builds trust in the system overall. Be transparent with users about what the system can and can't do reliably. We added confidence scores that explicitly flag when a result might involve negation, numerical comparison, or other potential weak points. Users appreciate the honesty, and it builds trust in the system overall. Here's the most important thing I've learnt:**these models don't understand language the way humans do – they understand statistical patterns. When I stopped expecting human-like understanding and started treating them as sophisticated pattern-matching tools with specific blind spots, my systems got better. Much better. The blind spots I've described aren't going away anytime soon – they're baked into how these models work. But if you know they're there, you can design around them. And sometimes, acknowledging a limitation is the first step toward overcoming it. Note: I have many more such cases found through experiments and I will not be covering them instead,I will start covering solution to each of the problems mentioned related to embeddings in my subsequent post. Note The next continuation article will be coming out soon. Stay tuned!! READ PART-1 here 
READ PART-2 here READ PART-1 here READ PART-1 here READ PART-1 here READ PART-2 here READ PART-2 here READ PART-2 here

Hallucinations by Design - (Part 3): Trusting Vectors Without Testing Them

Recent discussions at the Pacific Forum Economic Ministers Meeting have once again centered on the familiar themes of economic resilience, climate financing, and regional cooperation. While Pacific leaders continue to emphasize collaboration and unity as pathways to prosperity, we must ask ourselves a critical question i.e. Are we simply just repeating the same approaches that have yielded minimal results for decades? As Einstein allegedly said, "Insanity is doing the same thing over and over again and expecting different results." This wisdom perfectly captures the current state of economic development strategies in the Pacific region. The Traditional Approach - A History of Limited Success For decades, Pacific Island nations have been caught in cycles of dependency. Traditional economic resilience strategies have primarily focused on: Seeking external funding through aid and grants


Building facilities and institutions dependent on donor support


Creating top-down initiatives led by politicians with outdated mindsets


Relying on foreign expertise rather than developing local talent Seeking external funding through aid and grants Seeking external funding through aid and grants Building facilities and institutions dependent on donor support Building facilities and institutions dependent on donor support Creating top-down initiatives led by politicians with outdated mindsets Creating top-down initiatives led by politicians with outdated mindsets Relying on foreign expertise rather than developing local talent Relying on foreign expertise rather than developing local talent The recent Special Forum Economic Ministers Meeting in Tonga highlighted three primary initiatives i.e. the Pacific Roadmap for Economic Development (PRED), the Pacific Resilience Facility (PRF), and the Climate Finance Access & Mobilization Strategy (CFAMS). According to recent reports, the PRF treaty is set for ratification at the Pacific Islands Forum Leaders' Meeting in September 2025, with current pledges totaling approximately USD$137-150 million, with ambitious targets of reaching $250 million by next year and $1.5 billion in the coming years. While these initiatives sound promising on paper, they follow the same centralized, government-led approach that has historically delivered limited sustainable outcomes for ordinary Pacific Islanders. More troublingly, these massive funding figures mask a disturbing reality i.e. approximately 45-50% of foreign aid ultimately returns to development partners and their armies of consultants, with only about 50-55% actually absorbed by local communities. Even within that 50-55% that reaches recipient countries, additional portions may not directly benefit the most vulnerable populations due to administrative costs, corruption, or other inefficiencies within recipient countries' systems. It's also worth noting that aid distribution is uneven among recipient countries, with only 16.1% going to low-income countries that arguably need it most. Yet Pacific nations inherit infrastructure that future generations will be obligated, not by choice, to maintain, creating an ongoing cycle of dependency as operational expenses mount and more foreign aid is sought in subsequent years. Learning from Singapore's Transformation There's profound wisdom in the approach taken by Singapore's Lee Kuan Yew, who recognized that "our most valuable asset is in the ability of our people." Singapore's remarkable transformation from a resource poor island nation to a global economic powerhouse wasn't built on natural resources but on investing in its people. Lee emphasized that Singapore needed to "develop Singapore's only available natural resource, its people." This philosophy must become central to Pacific development strategies. The Pacific's greatest resource isn't aid money or facilities, it's the innovative potential and entrepreneurial spirit of its people. The Web3 Roadmap - A Path to True Sovereignty The emergence of Web3 technology offers a transformation pathway for Pacific communities to achieve financial freedom and sovereignty outside traditional power structures. Unlike previous digital evolutions, Web3 places ownership and control directly with users rather than centralized platforms. Web3's core promise is digital sovereignty, the ability for individuals and communities to control their digital assets, identity, and financial future without intermediaries. For Pacific communities, this means: Financial Inclusion Without Traditional Barriers: Through decentralized finance (DeFi), Pacific Islanders can access financial services regardless of location or traditional banking infrastructure.


Direct Value Creation: Content creators, artists, and knowledge holders can monetize their contributions without middlemen taking the majority of earnings.


Community-Led Governance: Decentralized Autonomous Organizations (DAOs) enable communities to make collective decisions about resource allocation and development priorities.


Resilient Financial Systems: Cryptocurrency offers protection against currency devaluation and enables cross-border transfers without traditional banking fees. Financial Inclusion Without Traditional Barriers: Through decentralized finance (DeFi), Pacific Islanders can access financial services regardless of location or traditional banking infrastructure. Financial Inclusion Without Traditional Barriers: Through decentralized finance (DeFi), Pacific Islanders can access financial services regardless of location or traditional banking infrastructure. Financial Inclusion Without Traditional Barriers Direct Value Creation: Content creators, artists, and knowledge holders can monetize their contributions without middlemen taking the majority of earnings. Direct Value Creation: Content creators, artists, and knowledge holders can monetize their contributions without middlemen taking the majority of earnings. Direct Value Creation Community-Led Governance: Decentralized Autonomous Organizations (DAOs) enable communities to make collective decisions about resource allocation and development priorities. Community-Led Governance: Decentralized Autonomous Organizations (DAOs) enable communities to make collective decisions about resource allocation and development priorities. Community-Led Governance Resilient Financial Systems: Cryptocurrency offers protection against currency devaluation and enables cross-border transfers without traditional banking fees. Resilient Financial Systems: Cryptocurrency offers protection against currency devaluation and enables cross-border transfers without traditional banking fees. Resilient Financial Systems The Pasifika Web3 Tech Hub initiative demonstrates this forward thinking approach. As outlined in our constitution (available at https://github.com/Pasifika-Web3-Tech-Hub/constitution), we are building a community driven ecosystem that empowers Pacific Islanders to participate in and benefit from the emerging Web3 economy without relying on traditional centralized institutions. https://github.com/Pasifika-Web3-Tech-Hub/constitution Our mission statement clearly states i.e. "To empower Pacific Islanders with the knowledge, skills, and infrastructure to participate in and benefit from the Web3 economy, promoting economic sovereignty, technological innovation, and cultural preservation." Rather than waiting for government led programs, this grassroots initiative aims to empower individuals across Pacific communities with the knowledge and tools to participate in the global digital economy. Emancipation from Mental Slavery True economic transformation in the Pacific requires more than new technologies, it demands a fundamental shift in mindset. We must move beyond the "asking for handout" mentality that has become embedded in regional development discourse. This means: Embracing individual responsibility for economic advancement
Questioning the effectiveness of top-down, politician led development models
Building local technical capacity rather than perpetual reliance on foreign expertise
Developing community owned digital infrastructure rather than systems controlled by external entities Embracing individual responsibility for economic advancement Questioning the effectiveness of top-down, politician led development models Building local technical capacity rather than perpetual reliance on foreign expertise Developing community owned digital infrastructure rather than systems controlled by external entities A New Vision for Pacific Economic Resilience Instead of another round of high level forums producing well meaning but ultimately ineffective roadmaps, we need a ground up revolution in how we approach development: Invest in Digital Literacy: Ensure all Pacific Islanders, especially youth, have the skills to participate in the digital economy.
Build Community Owned Infrastructure: Develop decentralized systems that keep control and profits within Pacific communities.
Create Regulatory Sandboxes: Establish innovation friendly regulatory environments that allow Pacific entrepreneurs to experiment with blockchain and Web3 technologies.
Promote Pacific Built Solutions: Support the development of applications designed specifically for Pacific contexts and challenges.
Protect Digital Sovereignty: Resist imposed digital frameworks from external powers and develop uniquely Pacific approaches to digital governance. Invest in Digital Literacy: Ensure all Pacific Islanders, especially youth, have the skills to participate in the digital economy. Invest in Digital Literacy Build Community Owned Infrastructure: Develop decentralized systems that keep control and profits within Pacific communities. Build Community Owned Infrastructure Create Regulatory Sandboxes: Establish innovation friendly regulatory environments that allow Pacific entrepreneurs to experiment with blockchain and Web3 technologies. Create Regulatory Sandboxes Promote Pacific Built Solutions: Support the development of applications designed specifically for Pacific contexts and challenges. Promote Pacific Built Solutions Protect Digital Sovereignty: Resist imposed digital frameworks from external powers and develop uniquely Pacific approaches to digital governance. Protect Digital Sovereignty The Fundamental Question - Building Sovereignty or Perpetuating Dependency? As Pacific nations continue pursuing traditional development pathways and funding mechanisms, we must ask a critical question i.e. Are we building a Pacific for the sovereignty of our future generations, or are we building future slaves to foreign aid? The current approach risks creating a perpetual dependency cycle where impressive funding announcements make headlines, but the lasting legacy is debt, maintenance obligations, and diminished sovereignty. Our leaders should be questioning whether these massive foreign led initiatives truly serve the long term interests of Pacific peoples or merely perpetuate neocolonial economic relationships. Conclusion - The Time for Change Is Now The path to true economic resilience and prosperity in the Pacific won't come from another round of ministerial meetings or external funding mechanisms. It will emerge when we recognize, like Singapore did, that our only true resource is our people, and when we empower those people with the tools, skills, and mindset to create their own financial freedom. Web3 technologies offer an unprecedented opportunity for Pacific communities to leapfrog traditional development pathways and create new models of prosperity. But seizing this opportunity requires us to break free from outdated thinking and embrace individual and community driven approaches to development. The future of the Pacific lies not in the hands of politicians or international organizations, but in the creative potential of each Pacific Islander equipped with the digital tools to build a sovereign financial future. The time for transformation is now. Disclaimer: The boomerang aid figures presented are estimates compiled from various 2020-2023 sources. Percentages vary by donor country, methodology, and reporting period. This information provides a general understanding rather than precise measurements and should not be considered definitive. For academic, policy, or professional use, please consult primary sources, official OECD reports, and peer-reviewed research. Disclaimer: The boomerang aid figures presented are estimates compiled from various 2020-2023 sources. Percentages vary by donor country, methodology, and reporting period. This information provides a general understanding rather than precise measurements and should not be considered definitive. For academic, policy, or professional use, please consult primary sources, official OECD reports, and peer-reviewed research.

Pacific Nations Are Tired of Playing the Aid Game. Web3 Might Be the Exit Strategy

At its core, data indexing is the process of transforming raw data into a format that's optimized for retrieval. Unlike an arbitrary application that may generate new source-of-truth data, indexing pipelines process existing data in various ways while maintaining traceability back to the original source. This intrinsic nature - being a derivative rather than a source of truth - creates unique challenges and requirements. Characteristics of a Good Indexing Pipeline A well-designed indexing pipeline should possess several key traits: 1. Ease of Building People should be able to build a new indexing pipeline without mastering techniques such as database manipulation/access, streaming processing, parallelization, fault recovery, etc. In addition, transformation components (a.k.a. operations) should be easily composable and reusable across different pipelines. 2. Maintainability The pipeline should be easy to understand, modify, and debug. Complex transformation logic should be manageable without becoming a maintenance burden. On the other hand, the indexing pipeline is a stateful system, so besides the transformation logic, it's also important to expose the clear status of the pipeline states, e.g., statistics of the number of data entries, their freshness, and how a specific piece of derived data tracks back to the original source. 3. Cost-Effectiveness Data transformation (with the necessary tracking of relationships between data) should be done efficiently without excessive computational or storage costs. Moreover, existing computations should be reused whenever possible. For example, 1% of document change, or a chunking strategy change that only affects 1% of chunks, shouldn't need to entail rerunning the expensive embedding model over the entire dataset. 4. Indexing Freshness For many applications, the source of truth for indexing is consistently updated, so it's important to make sure the indexing pipeline is also updated accordingly in a timely manner. Common Challenges in Indexing Pipelines Incremental Updates Are Challenging The ability to process only new or changed data rather than reprocessing everything is crucial for both cost efficiency and indexing freshness. This becomes especially important as your data grows. To make incremental update work, we need to carefully track the state of the pipeline, to decide which portion of the data needs to be reprocessed, and make sure states derived from old versions are fully deleted or replaced. It's challenging to make things right while considering various complexities, like fan-in/fan-out in transformations, out-of-order processing, recovery after early termination, etc. Upgradability Often Overlooked Many implementations focus on the initial setup but neglect how the pipeline will evolve. When requirements change or new processing steps need to be added, the system should adapt without requiring a complete rebuild. Traditional pipeline implementations often struggle with changes to the processing steps. Adding or modifying steps typically requires reprocessing all data, which can be extremely expensive and involves a manual process. The Deterministic Logic Trap Many systems require deterministic processing logic - meaning the same input should always produce the same output. This becomes problematic when: Entry deletion needs to be handled
Processing logic naturally evolves
Keys generated in previous runs don't match current runs, leading to data leaks Entry deletion needs to be handled Processing logic naturally evolves Keys generated in previous runs don't match current runs, leading to data leaks How CocoIndex Solves These Challenges CocoIndex approaches indexing pipelines with a fundamentally different mental model - similar to how React revolutionized UI development compared to vanilla JavaScript. Instead of focusing on the mechanics of data processing, users can concentrate on their business logic and desired state. CocoIndex Stateless Logic: Users write pure transformation logic without worrying about state management
Automatic Delta Processing: CocoIndex handles incremental updates efficiently
Built-in Trackability: Every transformed piece of data maintains its lineage to source
Flexible Evolution: On pipeline changes, past intermediate states can still be reused whenever possible
Non-Deterministic Friendly: With data lineage clearly tracked, even without determinism of processing logic, CocoIndex can still make sure stale states are properly purged Stateless Logic: Users write pure transformation logic without worrying about state management Stateless Logic Automatic Delta Processing: CocoIndex handles incremental updates efficiently Automatic Delta Processing Built-in Trackability: Every transformed piece of data maintains its lineage to source Built-in Trackability Flexible Evolution: On pipeline changes, past intermediate states can still be reused whenever possible Flexible Evolution Non-Deterministic Friendly: With data lineage clearly tracked, even without determinism of processing logic, CocoIndex can still make sure stale states are properly purged Non-Deterministic Friendly Subtle Complexities We Handle Managing processing state across pipeline updates
Ensuring data consistency during partial updates
Smooth recovery from early termination of the pipeline
Optimizing resource usage automatically
Maintaining data lineage and relationships Managing processing state across pipeline updates Ensuring data consistency during partial updates Smooth recovery from early termination of the pipeline Optimizing resource usage automatically Maintaining data lineage and relationships The Mental Model Shift Just as React changed how developers think about UI updates by introducing the concept of declarative rendering, CocoIndex changes how we think about data indexing. Instead of writing imperative processing logic, users declare their desired transformations and let CocoIndex handle the complexities of efficient execution. This shift allows developers to focus on what their data should look like rather than the mechanics of how to get it there. The result is more maintainable, efficient, and reliable indexing pipelines that can evolve with your application's needs. Finally, A well-designed indexing pipeline is crucial for production RAG applications, but building one that's maintainable, efficient, and evolvable is challenging. CocoIndex provides a framework that handles these complexities while allowing developers to focus on their core business logic. By learning from the challenges faced by traditional approaches, we've created a system that makes robust data indexing accessible to everyone building RAG applications. It would mean a lot to us if you could support CocoIndex on Github (https://github.com/cocoindex-io/cocoindex) with a star if you like our work. Thank you so much with a warm coconut hug 🥥🤗. https://github.com/cocoindex-io/cocoindex

Data Indexing and Common Challenges

The AI Industry’s Love Affair With Overengineering Needs an Intervention

Любовната връзка на индустрията с свръхинженерството се нуждае от намеса

Η ερωτική σχέση της βιομηχανίας AI με την υπερ-μηχανική χρειάζεται παρέμβαση

El amor de la industria de la IA con la overengineering necesita una intervención

رابطه عشقی صنعت هوش مصنوعی با مهندسی بیش از حد نیاز به مداخله دارد

AI արդյունաբերության սիրային կապը Overengineering-ի հետ միջամտության կարիք ունի

AI業界のオーバーエンジニアリングとの恋愛関係には介入が必要

កិច្ចការស្នេហារបស់ឧស្សាហកម្ម AI ជាមួយនឹង Overengineering ត្រូវការអន្តរាគមន៍

AI pramonės meilės reikalas su overengineering reikalauja intervencijos

AI industrijas mīlas dēkai ar pārdomātu inženieriju ir nepieciešama iejaukšanās

ERROR: invalid target_lang (sn)

Qalabka dhismaha iyo dhismaha dhismaha waxaa loo isticmaali karaa in ay ku saabsan dhismaha dhismaha.

Муҳаббати саноати AI бо муҳандисии аз ҳад зиёд ба дахолат ниёз дорад

Sentient has carved out a unique path, blending blockchain technology with open-source AI to create a community-owned ecosystem. With over 1 million waitlist sign-ups for Sentient Chat in just 24 hours and a record-breaking 650,000 NFT mint for their decentralized AI model, Dobby, Sentient is redefining the future of AI. Sentient At the helm of this revolution is Himanshu Tyagi, Co-founder of Sentient and a Professor at the Indian Institute of Science, whose expertise in AI and blockchain is driving innovations like the Reasoning Agent and a Perplexity-rivaling chatbot. We sit down with Himanshu to explore the technical challenges, groundbreaking solutions, and ambitious vision behind Sentient’s rise. Ishan Pandey: Sentient Chat has exploded onto the scene with over 1 million waitlist sign-ups in under 24 hours. What technical challenges did you face in building an AI platform that’s both decentralized and consumer-friendly? Ishan Pandey: Himanshu Tyagi: Decentralization is foundational to our mission, extending far beyond just Sentient Chat. We believe that the future of AI must be open and community-driven, as decentralization uniquely empowers active community participation. This philosophy cascades directly into our products—most notably, Sentient Chat. Himanshu Tyagi: With Sentient Chat, our goal is to ensure that the community truly feels ownership. Users aren't merely passive consumers; they are creators, molders, and stewards of the platform. They have full control—from the underlying AI model and agents, to data and governance—enabling them to shape the product precisely to fit their diverse needs and use cases. Critically, decentralization and usability reinforce each other in Sentient Chat. By empowering users to customize and enhance the platform, every innovation developed by one user or team becomes accessible to the broader community. This collective contribution enriches the platform's usability, continually driving improvements and innovations that benefit everyone involved. Ultimately, decentralization doesn't complicate the user experience—it enhances it, creating a vibrant ecosystem where collective creativity and collaboration translate directly into a more powerful, flexible, and user-friendly platform. Ishan Pandey: You’ve positioned Sentient Chat as a competitor to Perplexity, powered by Dobby, the world’s first community-owned AI model. Can you walk us through the technical innovation behind Dobby and how it supports Sentient Chat’s unique capabilities? Ishan Pandey: Himanshu Tyagi: AI is rapidly becoming the intermediary between information networks and users, granting it significant power to influence user perspectives based on inherent biases. Our goal is to empower communities by enabling them to align AI models according to their specific values, thus promoting a diverse ecosystem of "Loyal AI" models. While biases will always exist due to training data, users should not be limited to generalized models that simply average internet opinions. Instead, we envision users selecting from a wide array of models aligned with distinct community values. Himanshu Tyagi: Dobby represents the first of such models, explicitly aligned with the crypto community's values—pro-crypto and pro-personal freedom. With Dobby, we've achieved three notable technical breakthroughs: Effective Value Alignment Without Performance Loss: Aligning AI models to specific values through fine-tuning is notoriously challenging, often leading to degraded performance and incoherent outputs. With Dobby, we've successfully fine-tuned the Llama model to embrace specific values while maintaining excellent performance across critical benchmarks.


Human-centric Tone and Freedom of Expression with Safety: Achieving our desired alignment required removing existing guardrails from Llama and carefully rebuilding them. Surprisingly, this process enhanced the model’s tone, resulting in more natural and human-centric interactions. Even with relaxed constraints, Dobby remains safe, as demonstrated by metrics from our "Sorry Bench." Moreover, our Dobby-mini 8B and Dobby 70B models exhibit granular control over specific safety dimensions (such as hate speech, financial, medical, and legal advice, and explicit content), allowing us precise adjustments of the model's safety profile.


Fingerprinting for the Most Distributed Model Ownership Ever: Dobby is currently the most widely distributed model, with over 660,000 owners. This unprecedented level of distribution is facilitated by our innovative fingerprinting technology. Fingerprinting injects unique key-response pairs into the model, enabling robust verification of ownership. These fingerprints have a negligible impact on overall model performance and are resilient to removal through fine-tuning, model merging, or other modifications. Effective Value Alignment Without Performance Loss: Aligning AI models to specific values through fine-tuning is notoriously challenging, often leading to degraded performance and incoherent outputs. With Dobby, we've successfully fine-tuned the Llama model to embrace specific values while maintaining excellent performance across critical benchmarks. Effective Value Alignment Without Performance Loss: Aligning AI models to specific values through fine-tuning is notoriously challenging, often leading to degraded performance and incoherent outputs. With Dobby, we've successfully fine-tuned the Llama model to embrace specific values while maintaining excellent performance across critical benchmarks. Effective Value Alignment Without Performance Loss: Human-centric Tone and Freedom of Expression with Safety: Achieving our desired alignment required removing existing guardrails from Llama and carefully rebuilding them. Surprisingly, this process enhanced the model’s tone, resulting in more natural and human-centric interactions. Even with relaxed constraints, Dobby remains safe, as demonstrated by metrics from our "Sorry Bench." Moreover, our Dobby-mini 8B and Dobby 70B models exhibit granular control over specific safety dimensions (such as hate speech, financial, medical, and legal advice, and explicit content), allowing us precise adjustments of the model's safety profile. Human-centric Tone and Freedom of Expression with Safety: Achieving our desired alignment required removing existing guardrails from Llama and carefully rebuilding them. Surprisingly, this process enhanced the model’s tone, resulting in more natural and human-centric interactions. Even with relaxed constraints, Dobby remains safe, as demonstrated by metrics from our "Sorry Bench." Moreover, our Dobby-mini 8B and Dobby 70B models exhibit granular control over specific safety dimensions (such as hate speech, financial, medical, and legal advice, and explicit content), allowing us precise adjustments of the model's safety profile. Human-centric Tone and Freedom of Expression with Safety: Fingerprinting for the Most Distributed Model Ownership Ever: Dobby is currently the most widely distributed model, with over 660,000 owners. This unprecedented level of distribution is facilitated by our innovative fingerprinting technology. Fingerprinting injects unique key-response pairs into the model, enabling robust verification of ownership. These fingerprints have a negligible impact on overall model performance and are resilient to removal through fine-tuning, model merging, or other modifications. Fingerprinting for the Most Distributed Model Ownership Ever: Dobby is currently the most widely distributed model, with over 660,000 owners. This unprecedented level of distribution is facilitated by our innovative fingerprinting technology. Fingerprinting injects unique key-response pairs into the model, enabling robust verification of ownership. These fingerprints have a negligible impact on overall model performance and are resilient to removal through fine-tuning, model merging, or other modifications. Fingerprinting for the Most Distributed Model Ownership Ever: Sentient Chat is our AI-powered search platform, designed to showcase and provide users with a direct experience of the intangible concept of Loyal AI. This platform demonstrates the competitive edge of open-source ecosystems, successfully outperforming platforms like Perplexity and matching the search benchmarks of ChatGPT. Powered by Dobby, Sentient Chat illustrates how Loyal AI can directly influence the information delivered to users, presenting content concisely and engagingly. The integration of Dobby into Sentient Chat has notably improved the efficiency and entertainment value of search interactions. With Dobby's clear, human-like, no-nonsense communication style, users can quickly and effectively access the information they seek, highlighting the platform's distinct advantage in AI-driven search experiences. Ishan Pandey: One of the most intriguing aspects of Sentient Chat is its Reasoning Agent, equipped with tools like search, calculator, and think. How did you architect this agent, and what technical hurdles did you overcome to make reasoning a core strength? Ishan Pandey: Himanshu Tyagi: Recent advancements have shown that generating executable Python code for tool calling yields a significant boost in performance compared to conventional JSON-based approaches. In particular, LLM’s are inherently adept at compressing the action-space of tasks using code. This naturally prompted the use of code as the mode of expression for our reasoning agent. Sometimes LLM's write code that does not execute/run the way it is intended to, this is remedied by a Python interpreter which checks if the code generated executes or throws errors, if it throws errors, then it will keep regenerating the code (with the previous error traces) until it outputs satisfactory code. Himanshu Tyagi: Another caveat is that the LLM may generate functional code that does not do what it is intended to do (e.g. calls the calculator tool instead of searching); the reasoning agent is capable of digesting the output of the information and assessing whether it fits with, on a general level, what it expects from the particular code/tool call. We tested two different types of agents in the Open Reasoning Agent portion of the ODS framework: Chain-of-Thought with ReAct agent and Chain-of-Code with CodeAct agent. We tested two different types of agents in the Open Reasoning Agent portion of the ODS framework: Chain-of-Thought with ReAct agent and Chain-of-Code with CodeAct agent. In the Chain-of-Thought ReAct architecture, the agent follows an iterative reasoning-action loop. First, the 'Think' step allows the agent to reason internally, articulating plans or interpreting intermediate results in natural language. Whenever the agent encounters uncertainty or lacks critical information, it invokes the 'Search' tool, actively retrieving external information from the web. For arithmetic or computational tasks that require high precision, the agent utilizes the 'Calculate' tool. Thus, CoT-ReAct continuously cycles through thinking, searching, and calculating as needed, until it arrives at a reliable solution. On the other hand, the Chain-of-Code or CodeAct agent employs executable code to perform its reasoning tasks. Initially, the agent thinks through its strategy, similar to CoT-ReAct. However, instead of relying purely on natural language, it generates executable Python code as part of its reasoning. The 'Calculate' function is seamlessly embedded here, directly integrated into the Python execution environment. This allows for accurate computations and algorithmic reasoning. Like CoT-ReAct, CodeAct also uses the 'Search' tool when additional external information is necessary, incorporating the results directly into its code-generation and execution processes. The key difference lies in how each agent interacts with these tools. For CoT-ReAct, 'Think', 'Search', and 'Calculate' are distinctly separate actions explicitly chosen during reasoning. For CodeAct, while 'Think' remains an explicit planning step, 'Calculate' is inherently built into the code execution itself, making computational tasks integral rather than external. 'Search' remains similarly explicit in both approaches, providing vital external context whenever the agent's internal knowledge is insufficient. Ishan Pandey: Sentient completed a record-breaking 650,000 NFT mint for Dobby, tying ownership to a decentralized AI model. What technical infrastructure did you build to handle this scale, and how does blockchain enhance the community ownership aspect? Ishan Pandey: Himanshu Tyagi: This was not a technically challenging problem – modern blockchains can easily handle such load. The most interesting part here is that this is the first time anybody has demonstrated this scale of “direct democracy” in model governance. In centralized companies, this governance is done by a tiny alignment team which has sometimes even modified the model to match its own beliefs. Anthropic had done some simple experiments under the name of constitutional AI with about 1000 people, but nothing close to this open governance by community. We wanted to give this power to the people and blockchains are perfect instruments for that. Dobby community already decided what kind of persona Dobby should have, leashed or unhinged (they chose unhinged over leashed). In future, all decisions about Dobby-unhinged model will be taken by this community, who will also get rewarded as model usage increases. Himanshu Tyagi: Going forward, these ownership NFTs will be converted to ownership tokens on our blockchain which can be used to govern alignment and updates to this model. Furthermore, the associated fingerprints will be committed as Merkle root to the same contract – anyone can verify the identity of the model by asking our fingerprint queries, and can check that the same queries are committed with the ownership contract. Ishan Pandey: Your Reasoning Agent integrates real-time tools and aims to tackle ambiguous, subjective tasks. How does this differ from existing reasoning agents, like OpenAI’s Deep Research, and what novel solutions did you implement to push the boundaries of AI reasoning? Ishan Pandey: Himanshu Tyagi: When comparing our open-source platform to closed-source competitors, the primary competitive advantage of closed systems has traditionally been their technology. However, platforms like Sentient Chat and Open Data Sources (ODS) have effectively leveled the playing field by democratizing AI knowledge, making our framework completely open to the community. Closed-source platforms, such as those by OpenAI, operate as black boxes; neither you, I, nor anyone else can fully comprehend their internal processes. Himanshu Tyagi: In contrast, our open framework invites continuous innovation from the wider community, enabling rapid identification and resolution of gaps that might otherwise remain unnoticed. This collaborative, open-input and open-output approach is key to building the best possible search platform. Our ultimate vision includes enabling users to contribute their own data sources and agents, thereby drastically enhancing Sentient Chat and ODS through collective, community-driven improvements. Ishan Pandey: You’ve emphasized Sentient Chat’s edge over competitors like Perplexity with its 15+ AI agent integration. What technical challenges did integrating multiple agents into a single chatbot interface present, and how did you solve them? Ishan Pandey: Himanshu Tyagi: Technically, we are addressing many complex software and design challenges to allow a broad variety of agents to participate in a singular user experience through Sentient Chat. We are also offering many tools to these agent builders such as our state-of-the-art AI search APIs, the Sentient Secure Enclave offering for building “unruggable agents,” etc. But more than addressing these technical challenges, the most interesting part here is the possibility of a completely new way to access internet knowledge. Himanshu Tyagi: For decades, Google has dominated search by focusing primarily on helping users find information online. Google's reliance on advertising revenue, rooted in recommending sources for information, inherently limits its ability to innovate beyond traditional information retrieval. AI, on the other hand, provides an opportunity to bypass mere information gathering, enabling users to directly execute tasks without needing separate analysis and action phases, which is our exact competitive advantage against the behemoth of Google. This ambitious vision requires a diverse ecosystem—one that integrates varied indexed data sources with numerous AI agents capable of performing specialized actions. The key to unlocking this potential lies in creating a transparent and open platform, actively incentivizing broad community participation. Data providers must clearly understand the value their contributions bring, while agent developers must seamlessly integrate their services to enhance overall functionality. Such an ecosystem must be community-governed, ensuring fairness, openness, and continual innovation driven by collective contribution and collaboration. By the way, a lot more than 15 agents are coming on Sentient Chat! Ishan Pandey: Looking ahead, what’s next for Sentient in terms of technical innovation? How do you plan to evolve the Reasoning Agent and expand the open AGI ecosystem? Ishan Pandey: Himanshu Tyagi: We see a future where AI becomes an intermediary between all human senses, global information, and social networks. Sentient’s vision is to make sure that this AI is loyal to us. This singular objective drives all our development. For reasoning agents, we want to make sure that developers get access to models whose reasoning capabilities and skills are aligned with their interests or the use case. Himanshu Tyagi: For instance, if you are building a sovereign agent that can control wallets, the underlying models should have reasoning capabilities to ensure that malicious actors cannot manipulate prompts and steal funds. This is alignment – we will be building such aligned models. Every developer building in the Open AGI ecosystem should have the option of using AI models that will be loyal to their application or users. Sentient Chat is the collective offering of all such agents built on Loyal AI. Don’t forget to like and share the story! Vested Interest Disclosure: This author is an independent contributor publishing via our business blogging program. HackerNoon has reviewed the report for quality, but the claims herein belong to the author. #DYO Vested Interest Disclosure: This author is an independent contributor publishing via our business blogging program. HackerNoon has reviewed the report for quality, but the claims herein belong to the author. #DYO Vested Interest Disclosure: business blogging program business blogging program

How Sentient’s Reasoning Agent is Outsmarting the Competition: Inside Look 

كيف يعمل عميل التفكير في Sentient على تجنب المنافسة: النظر في الداخل

Com l'agent de raonament de Sentient està superant la competència: a l'interior

Cómo el agente de razonamiento de Sentient está superando a la competencia: Inside Look

Bagaimana Agen Penalaran Sentient Mengalahkan Pesaingnya: Pandangan dari Dalam

ERROR: invalid target_lang (iw)

Sentient の推論エージェントが競合他社を凌駕する方法: 内部の分析

Како резонантниот агент на Sentient ја надмудри конкуренцијата: погледнете одвнатре 

Bagaimana Sentient's Reasoning Agent Mengabaikan Persaingan: Melihat Dalam

Imaynatachus Sentient’s Reasoning Agent atipanakuymanta aswan yachayniyuq kachkan: Ukhupi qhaway 

Как рассуждающий агент Sentient вытесняет конкуренцию: Внутренний взгляд

Sida Wakiilka Sababta Sentient uu uga Wanagsan yahay Tartanka: Gudaha Fiiri 

Sentient的推理代理如何超越竞争:内部看法

Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2 Data 2.1 Data Sources 2.1 Data Sources 2.2 SS and SI Categories 2.2 SS and SI Categories 3 Methods 3.1 Lexicon Creation and Expansion 3.1 Lexicon Creation and Expansion 3.2 Annotations 3.2 Annotations 3.3 System Description 3.3 System Description 4 Results 4.1 Demographics and 4.2 System Performance 4.1 Demographics and 4.2 System Performance 5 Discussion 5 Discussion 5.1 Limitations 5.1 Limitations 6 Conclusion, Reproducibility, Funding, Acknowledgments, Author Contributions, and References 6 Conclusion, Reproducibility, Funding, Acknowledgments, Author Contributions, and References SUPPLEMENTARY SUPPLEMENTARY Guidelines for Annotating Social Support and Social Isolation in Clinical Notes Guidelines for Annotating Social Support and Social Isolation in Clinical Notes Other Supervised Models Other Supervised Models 3.2 Annotations To create gold standard data for developing the NLP pipelines, we selected 300 notes from 300 unique patients at MSHS and 225 notes from 221 unique patients at WCM for fine- and coarsegrained manual annotation. Notes were chosen from unique patients to maximize the contextual diversity of SS/SI terms (different note-writers, different time periods, and avoiding redundancy caused by copy-forward practices within a single patient’s EHR). To optimize the gold standard annotation set of notes, those selected for review were enriched for mentions of SS and SI: 75 notes were selected that had at least one occurrence of an SI lexicon term, another 75 notes for SS, and finally 75 notes were randomly selected from the reminder of underlying corpus. At MSHS, 75 additional notes were selected that contained a clinical note template to further enrich the annotation corpus for notes in which a clinician was prompted (by the template) to assess SS/SI. The Brat Rapid Annotation Tool (BRAT) [40] was used to annotate the notes manually with the same annotation configuration schema across sites. The annotation guideline‡ and lexicons are provided in Supplementary Tables S3. Initially, the annotations were performed at the entity level (every instance of a lexicon term in the note text) using BRAT. For evaluation, the entity-level annotations were converted to “document” (note) level. For example, if there was a single entity mentioning loneliness and two mentions of instrumental support in a given note, the loneliness and instrumental support subcategories were assigned to that note. Finally, the coarse-grained categories were assigned to each document using rules. SS was assigned to a document if there were one or more mentions of any SS subcategories and similarly, SI was labeled if there were one or more mentions of any SI subcategories. The above note would be annotated with both SI (for loneliness) and SS (for instrumental support). The notes were meticulously reviewed by two annotators and disagreements were resolved by a third adjudicator to create the final gold-standard corpus. For coarse-grained annotation, the inter-annotator agreement (IAA) Cohen’s Kappa scores were 0.92 [MSHS] and 0.86 [WCM]; for fine-grained, 0.77 [MSHS] and 0.81 [WCM]. The counts of fine- and coarse-grained categories found in the gold-standard data are provided in Supplemental Table S5. The rule book was used to train the annotators and were continually updated during the adjudication process. Often, disagreeing annotations could both be seen as correct given the inherent subjectivity of the classification process; however, new rules were created to arrive at one consistent label for edge cases. Sometimes, rules were created for more practical reasons, for example, mentions of ‘psychotherapy’ were excluded from emotional support because otherwise almost every note in the MSHS psychiatric corpus would be flagged. Of note, mentions were only labelled when SS/SI was explicit and not implied. For example, a mention of ‘boyfriend’ or ‘living alone’ without further context would not count. The general subcategory became a “catch-all” for mentions that clearly involved support or isolation, but a single fine-grained category could not be discerned. For example, ‘staying on his best friend’s couch’ could be seen as instrumental support (providing shelter), social network (having friends), or emotional support (best friend implies a level of closeness). At both institutions, the IAA was reflective of the subjective, overlapping nature of the fine-grained subcategories. Another reason for disagreements between annotators was the site-specific familiarity required to recognize acronyms and social services, e.g., ‘HASA stands for the HIV/AIDS Services Administration.’ This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv ‡ rule book and annotation guideline are used interchangeably Authors:
(1) Braja Gopal Patra, Weill Cornell Medicine, New York, NY, USA and co-first authors;
(2) Lauren A. Lepow, Icahn School of Medicine at Mount Sinai, New York, NY, USA and co-first authors;
(3) Praneet Kasi Reddy Jagadeesh Kumar. Weill Cornell Medicine, New York, NY, USA;
(4) Veer Vekaria, Weill Cornell Medicine, New York, NY, USA;
(5) Mohit Manoj Sharma, Weill Cornell Medicine, New York, NY, USA;
(6) Prakash Adekkanattu, Weill Cornell Medicine, New York, NY, USA;
(7) Brian Fennessy, Icahn School of Medicine at Mount Sinai, New York, NY, USA;
(8) Gavin Hynes, Icahn School of Medicine at Mount Sinai, New York, NY, USA;
(9) Isotta Landi, Icahn School of Medicine at Mount Sinai, New York, NY, USA;
(10) Jorge A. Sanchez-Ruiz, Mayo Clinic, Rochester, MN, USA;
(11) Euijung Ryu, Mayo Clinic, Rochester, MN, USA;
(12) Joanna M. Biernacka, Mayo Clinic, Rochester, MN, USA;
(13) Girish N. Nadkarni, Icahn School of Medicine at Mount Sinai, New York, NY, USA;
(14) Ardesheer Talati, Columbia University Vagelos College of Physicians and Surgeons, New York, NY, USA and New York State Psychiatric Institute, New York, NY, USA;
(15) Myrna Weissman, Columbia University Vagelos College of Physicians and Surgeons, New York, NY, USA and New York State Psychiatric Institute, New York, NY, USA;
(16) Mark Olfson, Columbia University Vagelos College of Physicians and Surgeons, New York, NY, USA, New York State Psychiatric Institute, New York, NY, USA, and Columbia University Irving Medical Center, New York, NY, USA;
(17) J. John Mann, Columbia University Irving Medical Center, New York, NY, USA;
(18) Alexander W. Charney, Icahn School of Medicine at Mount Sinai, New York, NY, USA;
(19) Jyotishman Pathak, Weill Cornell Medicine, New York, NY, USA. Authors: Authors: (1) Braja Gopal Patra, Weill Cornell Medicine, New York, NY, USA and co-first authors; (2) Lauren A. Lepow, Icahn School of Medicine at Mount Sinai, New York, NY, USA and co-first authors; (3) Praneet Kasi Reddy Jagadeesh Kumar. Weill Cornell Medicine, New York, NY, USA; (4) Veer Vekaria, Weill Cornell Medicine, New York, NY, USA; (5) Mohit Manoj Sharma, Weill Cornell Medicine, New York, NY, USA; (6) Prakash Adekkanattu, Weill Cornell Medicine, New York, NY, USA; (7) Brian Fennessy, Icahn School of Medicine at Mount Sinai, New York, NY, USA; (8) Gavin Hynes, Icahn School of Medicine at Mount Sinai, New York, NY, USA; (9) Isotta Landi, Icahn School of Medicine at Mount Sinai, New York, NY, USA; (10) Jorge A. Sanchez-Ruiz, Mayo Clinic, Rochester, MN, USA; (11) Euijung Ryu, Mayo Clinic, Rochester, MN, USA; (12) Joanna M. Biernacka, Mayo Clinic, Rochester, MN, USA; (13) Girish N. Nadkarni, Icahn School of Medicine at Mount Sinai, New York, NY, USA; (14) Ardesheer Talati, Columbia University Vagelos College of Physicians and Surgeons, New York, NY, USA and New York State Psychiatric Institute, New York, NY, USA; (15) Myrna Weissman, Columbia University Vagelos College of Physicians and Surgeons, New York, NY, USA and New York State Psychiatric Institute, New York, NY, USA; (16) Mark Olfson, Columbia University Vagelos College of Physicians and Surgeons, New York, NY, USA, New York State Psychiatric Institute, New York, NY, USA, and Columbia University Irving Medical Center, New York, NY, USA; (17) J. John Mann, Columbia University Irving Medical Center, New York, NY, USA; (18) Alexander W. Charney, Icahn School of Medicine at Mount Sinai, New York, NY, USA; (19) Jyotishman Pathak, Weill Cornell Medicine, New York, NY, USA.

This Is How We Created Gold Standard Data for Developing the NLP Pipelines

Music & ML: Clustering on Spotify Audio Features

ERROR: invalid target_lang (ay)

Muzika i ML: grupisanje na Spotify audio funkcijama

Música i ML: agrupació a les funcions d'àudio de Spotify

Μουσική & ML: Συγκεντρώνοντας στο Spotify Audio Features

Música y ML: Clustering en características de audio de Spotify

Musique & ML: Clustering sur les fonctionnalités audio de Spotify

音楽と機械学習: Spotify オーディオ機能のクラスタリング

음악 및 ML: Spotify 오디오 기능 클러스터링

Musique & ML: Clustering ya Spotify Audio

Musika at ML: Pag-cluster sa Spotify Audio Features

ERROR: invalid target_lang (ts)

A Memory Hack That Makes AI Smarter With Sequences

Rules, Exceptions, and Exploration: The Secret to EXPLORER’s Success

ChatGPT Goes Ghibli, Google Gets Smarter, and Microsoft Embeds Knowledge at Scale

Recently, I made a comment about the idea of there being a “best” monitoring tool: Recently In fact, let’s get this out in the open: There simply isn’t a singular “best” monitoring tool out there any more than there’s one singular “best” programming language, or car model, or pizza style.* There isn’t a single tool which will cover 100% of your needs in every single use case. In fact, let’s get this out in the open: There simply isn’t a singular “best” monitoring tool out there any more than there’s one singular “best” programming language, or car model, or pizza style.* There isn’t a single tool which will cover 100% of your needs in every single use case. The comment got some pushback, both privately and in a few forums, so I wanted to dig into what I meant and why I felt that way. But before I do that, I want to set the record straight: I stand by what I said about there being no “best” monitoring tool. But I was flat-out lying about the other stuff. The hills I’m willing to die on are: The best programming language is Perl
The best car is the 1967 Ford Mustang 390 GT/A
The best pizza style is deep dish, and I’m partial to getting it from Tel Aviv Kosher Pizza in Chicago The best programming language is Perl The best car is the 1967 Ford Mustang 390 GT/A The best pizza style is deep dish, and I’m partial to getting it from Tel Aviv Kosher Pizza in Chicago Tel Aviv Kosher Pizza in Chicago With that cleared up, let’s get back to monitoring and observability. Zoom Zoom What really got me thinking about the false concept of a “best” monitoring tool was a video my son shared with me, comparing a Lucid Air Sapphire, a Bugatti Chiron, and a Tesla Plaid to see which was the fastest production car of all time. really a video my son shared with me Disclaimer: I am NOT a car person. My son Kaleb (who is 21, in his 2nd year of university to become a mechanical engineer, and on two different Baja SAE teams) very much is. Baja SAE After watching the video, Kaleb pointed out that if the race (which was on a quarter-mile track) had been a half-mile instead, the Bugatti would have won, hands down. This was because (reasons. I honestly couldn’t follow the things he was saying at this point. I leave it to the reader to imagine Star-Trek-like technobabble) My point in all this is that “best” – the fastest car in this case – is highly subject to other variables. The type of track (this was on a track that had been pre-treated with VHT, which makes it super sticky and affects traction), the distance, and even things like altitude and weather – these all can impact the ultimate outcome. But I’m talking about more than external factors. What is “best” can be affected by the ultimate use case. Cost is the easiest one that comes to mind. Yes, a Bugatti might be the fastest. But, possibly the “best” car is a Honda Civic because you value cost and reliability over speed. Or perhaps a Kia Sedona might be “best” because you need more seats. Or a Ford F150. Or a Ryder 16-wheeler. This all relates to monitoring and observability in ways that are both important and, sadly, novel to a lot of IT practitioners. We get so caught up in the “speeds and feeds” aspect of tools and solutions – how many flows per second, how many traces per collector, maximum ingest before backpressure occurs – that we often fail to stop and say, “Do I need that? Will I ever need that?” ever I shared this with my friend Kevin Sparenberg (another car guy, who writes occasionally on his blog: blog.kmsigma.com), He added: his blog: blog.kmsigma.com Monitoring tools are like any other tool. You have your favorite hammer, but it’s not appropriate in every scenario. You don’t (or shouldn’t) use a sledge for setting a nail, nor would you use a claw hammer for forging. You might have a favorite, but you need to consider the type of work you need accomplished. Monitoring tools are like any other tool. You have your favorite hammer, but it’s not appropriate in every scenario. You don’t (or shouldn’t) use a sledge for setting a nail, nor would you use a claw hammer for forging. You might have a favorite, but you need to consider the type of work you need accomplished. (Besides being a car guy, he’s a home-repair weekend warrior. This is just one of the many reasons why we’re friends.) “Best” Isn’t Always Best Part of the blame can be placed at the feet of vendors. I’ve worked at a few, and it’s rare to find one that has the tools to help customers quantify volume and cost before implementation. Most simply say, “let’s just get this installed and we’ll see where it lands and we can tune from there.”, blithely ignoring the way the level of effort (not to mention political maneuvering in the C-suite) to implement a new tool makes sunk-cost fallacy a near-certainty. But that’s only part of the blame. The other part rests at our feet – the monitoring engineers who need to better shoulder the responsibility of understanding and speaking for the needs of our organization. Because if we don’t, who will? Much of this comes back to things I’ve already ranted about: If we don’t have a plan for the monitoring and observability data being collected, any cost is going to seem to be too much. If you have a plan, you’ll know exactly how much the data is worth and be able to evaluate the cost of a tool. If we don’t have a plan Learn to speak the language of business, to frame things NOT in the technical terms that you find familiar and comfortable, but in terms that make it clear to the business why a new tool is needed and the value it will provide. Learn to speak the language of business Solve the problems your organization is actually having. Once again, it’s easy to get swept up by a vendor’s vision. But if your company isn’t having any of the problems that vendor vision describes, it’s all wasted time and money. Solve the problems your organization is actually having Kevin added another nuance. Hyper-focusing on a single metric isn’t just sloppy, it can lead to real problems down the road: I would even go so far as to revisit that zero-to-60 metric. What about zero-to-60-to-zero? Sure any car can get to 60, but which car is able to do that AND return to status quo quickly. If you need a good “bad” example, look at the Gen 1 for Dodge Viper. As my dad said, “plenty of giddy-up, virtually no whoa.” I would even go so far as to revisit that zero-to-60 metric. What about zero-to-60-to-zero? Sure any car can get to 60, but which car is able to do that AND return to status quo quickly. If you need a good “bad” example, look at the Gen 1 for Dodge Viper. As my dad said, “plenty of giddy-up, virtually no whoa.” How does that translate to monitoring and observability? Think about alerting. LOTS of tools are able to detect and trigger alerts based on extremely specific (and sensitive) triggers. But far fewer have the controls to detect and stop alert storms. My point in all this is to remember that “best” always has to be weighed against YOUR values: Your real-world, actual business needs
Your cost-to-benefit ratio
Your team’s skillsets
Your tolerance for toil and effort during the transition
Your willingness to support one more tool in perpetuity
…and so on. Your real-world, actual business needs Your cost-to-benefit ratio Your team’s skillsets Your tolerance for toil and effort during the transition Your willingness to support one more tool in perpetuity …and so on. Good Enough is Usually Good Enough There is a story that is equal parts old, hilarious, and fake. It involves a hapless young man (it’s ALWAYS a dude) who decides to mount a JATO rocket to his car to see just how fast he can go. And in the ensuing chaos, this young man (supposedly) earned himself a posthumous Darwin Award. decides to mount a JATO rocket to his car Darwin Award (Once again, I have to emphasize that this story is 100% fake and was even debunked on the very first episode of MythBusters) (Once again, I have to emphasize that this story is 100% fake and was even debunked on the very first episode of MythBusters) (Once again, I have to emphasize that this story is 100% fake and was even debunked on the very first episode of MythBusters) the very first episode of MythBusters However, one of the aspects of the story that makes it funny (at least in my opinion) is the sheer ridiculousness of it all. Sure, lots of folks want a fast car. Many of those folks are willing to spend a little extra for a car that’s a little faster than the norm. Fewer (but not zero) people might also be willing to go to great lengths to acquire not only a fast car but “the fastest” car. But strapping a rocket to the top of an old car – one that was clearly never intended to be used this way? That is some serious, janky, automotive slapstick. It tickles our funny bone by evoking those grainy sepia-tinted images of early 20th-century “flying machines” that were nothing more than 2 umbrellas strapped to a piston. As the final image of the JATO story fades in our mind, we can almost see the end card saying, “You just couldn’t leave well-enough alone, could you?” strapping a rocket to the top of an old car strapping a rocket to the top of an old car grainy sepia-tinted images Likewise, we need to foster a habit of self-restraint and technical reflection in our monitoring discipline. We have to recognize when our excitement about a tool’s ability to process 8 million log messages a second clouds our own ability to step back and say, “Why do I even HAVE 8 million log messages a second?” And if there’s a good reason for that volume, follow up with the question “Why do I need to send every single one of those messages across the internet to a vendor’s storage?” I’m not saying there’s nobody in the world who might need that. You might have a good reason. I just want to suggest you take a second to consider things before you end up creating your own JATO-powered observability disaster. The Solution is Both-And, not Either-Or. This is the thing most vendors don’t say, at least admit within earshot of investors and members of the board, because it flies in the face of the marketing hype and sales pitches they’ve worked so hard to craft. You need more than one monitoring and observability solution. You need to decide which systems (and more fundamentally, which data on those systems) use each tool. You’re going to have to split your budget (time, effort, skills, money) between those tools. You need more than one monitoring and observability solution. You need more than one monitoring and observability solution. This is an ugly but unavoidable truth. Over 27 years of working with monitoring and observability tools, the number of times I’ve seen a company that had exactly one monitoring solution is: zero. Even the ones that insist they do, after a little digging, have at least a few pockets of the environment that use something else, whether that’s a class of system (mainframes, minis, Windows NT servers, 6509’s); an organization or team that just went their own way or was acquired but never fully integrated; or a location that – due to their distance from the main organization, either functionally or geographically – has to maintain their own set of tools. And most orgs don’t have “at least a few pockets”. They have a full suite of overlapping solutions. You are going to have – more likely, you already DO have – multiple monitoring solutions in place. This is one of those tech realities that is simple but not easy – like supporting more than one operating system (whether servers or desktops), moving from branch-based development to feature flags, or building a multi-language, multi-cloud app. It’s the cost of doing real business in the real world. Observability, like life itself, is messy. It also, to quote Ian Malcom, finds a way. So will you. Here’s how: Plan and prepare to identify data types based on complex criteria: it might be a combination of location, system type, data type, and even time frame. Expect that, based on those parameters, you’ll then filter and transform the data before sending it in the correct direction. Also expect that some data types are so valuable, you’ll end up sending the same data in more than one direction. But also prepare to put boundaries in place so you aren’t doing that all the time because that gets expensive fast. that gets expensive fast As I said, you need to be ready to support multiple tools, but you should have a plan in place for how you’ll keep track of those tools, and identify their primary use case, and even set boundaries on the things they will NOT be permitted to monitor, so your stable of solutions doesn’t explosively get out of control. One way to do that is to set, for every data type or use case, a definitive choice for a primary tool that handles that data and a secondary one that you use as a gut-check. For larger organizations or more important data sets, you might have a tertiary, but draw the line there. Another way (complimentary to the first) is to understand that some tools are cheap and do a lot of things mostly ok, so you can spread them like peanut butter across the enterprise, while others are expensive (in time, effort, or money) and only do certain things well. So you should spread THOSE like caviar – only on the systems where they’ll do the most good. Finally, differentiate between management tools that also have a monitoring component and true monitoring and observability solutions. You shouldn’t get rid of management tools, but you also shouldn’t make them the primary source of truth for enterprise monitoring information because they are usually so vendor or system-specific, and it will lead (again) to that explosion of tools I cautioned against earlier. A Brief Buyer’s Guide A natural question to ask next is, “How do I know WHICH tools to get?” Once again, my buddy Kevin has some wise observations: Is it even worth mentioning bake-offs? Do people even do that anymore? Maybe tool A has these features, but tool B has these other ones. And they both share some other capabilities. We need all of it, but can’t get it in one package.
There’s nothing wrong with that. Leon’s point about having multiple tools is on target. Bias your decisions to picking the right tool for the right job (but at the same time, try not to collect too many tools. This ain’t Pokémon).Don’t buy for “this tool has this neat feature we don’t need, but maybe someday we’ll want.”. Buy (or deploy) for what you need now and in the near future. IT (and the business) is always growing and evolving. What you THINK is important today may be irrelevant in six months. Hopefully not, but thinking too far ahead – the infamous “five year plan” is just a waste of your effort and time. Is it even worth mentioning bake-offs? Do people even do that anymore? Maybe tool A has these features, but tool B has these other ones. And they both share some other capabilities. We need all of it, but can’t get it in one package. There’s nothing wrong with that. Leon’s point about having multiple tools is on target. Bias your decisions to picking the right tool for the right job (but at the same time, try not to collect too many tools. This ain’t Pokémon). Don’t buy for “this tool has this neat feature we don’t need, but maybe someday we’ll want.”. Buy (or deploy) for what you need now and in the near future. IT (and the business) is always growing and evolving. What you THINK is important today may be irrelevant in six months. Hopefully not, but thinking too far ahead – the infamous “five year plan” is just a waste of your effort and time. Taking a Victory Lap The point of this blog is pretty simple: There’s no such thing as “best”, and that goes for everything from cars to programming languages all the way to observability solutions. But more essential is my point about WHY there isn’t a single, specific “best” – it’s because context matters. Use case matters. To be more nuanced, there probably is a “best,” but what is best is extremely particular to you and your circumstances. So, the lesson in all this is to make sure you are clear about those circumstances and that you’re always weighing them against the “we’re the best” marketing hype you’ll hear from many vendors.

There Is No Such Thing as the Best Observability Tool

Deploying Transformers in Production: Simpler Than You Think

Disclaimer: I can't draw, so I used AI to create this: a robot (disguised as a cute creature from Ghibli's world) killing the next budding artist. Disclaimer: I can't draw, so I used AI to create this: a robot (disguised as a cute creature from Ghibli's world) killing the next budding artist. Disclaimer: If you compare the written work that AI generated three years ago to what ChatGPT generated last year, the difference is mind-boggling. And, if you compare artwork that AI generated six months ago to what can be generated today, the difference is… well, what can I say, nothing short of revolutionary.When OpenAI launched its new ChatGPT image generation feature, it took just two days for social media to explode with Ghibli-like renditions of everything from celebrities to memes to personal photos. Now, despite the tool being capable of generating images in a variety of art styles, why did people choose Ghibli? Studio Ghibli has a very distinct art style, thanks to its co-founder Hayao Miyazaki’s approach, which focuses on hand-drawn techniques. Frames are hand-drawn by artists (like Kazuo Oga), and all characters possess traits shaped by the human touch of the artist. It’s an unmistakable style, one that evokes a sense of nostalgia, innocence, and beauty—an experience that connects with people on a deeply emotional level. Anyone who has watched a Miyazaki film can immediately identify the Ghibli style, and it’s precisely that recognition that makes it so magical when people see themselves, or even memes, rendered in that familiar world. But not everyone is happy about this trend… Some art lovers and creators have voiced concerns, and for good reason. The implications of AI-generated art bother people, but for anyone who’s been paying attention, the question of whether AI will destroy artists and the livelihoods of people isn’t one of "if," but "when." Your blood might be boiling, but I am already in the fifth stage of grief (acceptance) and have already given up! So, I think it’s okay to generate images in the Studio Ghibli style to give yourself some joy because there will NEVER be another Ghibli or Pixar. If we consider what we have today and what might come in the next 5 or 10 years, it’s more than clear that there’s no room for the next budding artist with a unique sense of creativity.A career in any creative field is now far too risky, and there’s simply no room for a learner to progress into greatness or for a great artist to be well-funded when "good enough" can be produced at a fraction of the cost. NEVER be another Ghibli or Pixar As AI continues to advance, you would have to be a genius from the start to secure work in many fields. There will barely be any room for an entry or average-level writer, coder, designer, photographer, artist, or model to find decent work. Right now, AI is already capable of producing work at about 80% of an average human’s quality level, and it can do so at a fraction of the cost. Imagine where we’ll be in five years. Most businesses will likely choose AI-generated work that is a 7.5/10, given its significantly lower cost, rather than paying a premium for a 10/10 human-created version when the difference is often indistinguishable to the average person. Let’s take a moment to reflect on the sheer amount of time and resources that went into creating a single scene in a Studio Ghibli film. A four-second scene from The Wind Rises (2013) took an animator one year and three months to complete. Why was so much time invested? Because it was a Ghibli film, and such dedication is what defines the studio’s legacy. But this level of investment would never be spent on the next budding artist, not when AI can deliver good enough results far quicker and cheaper. Again, don’t focus on what we have today, consider where we’ll be in five years. A career in art will be extremely risky for the next generation, as most businesses won’t pay you if you’re anything less than above average.Hobbyists will keep the creative spark alive, but art as a career? That’s simply not worth doing. No business would fund the creativity or uniqueness of an unknown artist at the cost of profit margins when AI can deliver decent results for a small fraction of the cost. I’m not blaming businesses for this; it’s simply a matter of economics. Investing resources in an artist in the hope that they might one day become the next great is no longer a sound strategy. There might be a criticism of capitalism here but that is a whole different story. And for those still in denial, AI doesn't need to be better than a skilled human; it just needs to be good enough for your clients not to care. Plus, not all humans are equally skilled anyway.If you love art, whether as a creator or a viewer, embrace the last hurrah and the joy that Studio Ghibli and its style bring us, because there will NEVER be another great artist in the future. You might disagree, but I will wholeheartedly generate a few images as I process what a historic moment this is for art, knowing things will never be the same. Even if you don’t participate, someone else will, and there’s no stopping this. For me, this is a feeling of melancholy as we admire groundbreaking human art styles, perhaps for the very last time. TL;DR: AI exists to widen the gap between the labor class and the wealthy while eliminating jobs. Protecting big studios is the least of our concerns now. The genie can't be put back in the bottle. After visualizing what’s to come in the next five years, it’s better to accept and embrace this final celebration of art as the next Miyazaki ceases to exist. TL;DR: AI exists to widen the gap between the labor class and the wealthy while eliminating jobs. Protecting big studios is the least of our concerns now. The genie can't be put back in the bottle. After visualizing what’s to come in the next five years, it’s better to accept and embrace this final celebration of art as the next Miyazaki ceases to exist. TL;DR: Postscript Postscript My take on some very obvious criticisms and questions that might arise from this article. “Whoever creates this stuff has no idea what pain is whatsoever. An insert to life itself”??? “Whoever creates this stuff has no idea what pain is whatsoever. An insert to life itself”??? While I truly believe tech companies are wrong for stealing everyone’s work, including Miyazaki’s, it would also be wrong to sensationalize Miyazaki’s quote out of context. He made this comment after watching a demo of a horrifying, zombie-like creature with strange movements generated through AI in 2016. He also said the grotesque zombie movements reminded him of friends with disabilities and he thought that was an insult.  Clearly, his comments were specifically in the context of that demo and cannot be taken as his stance on generative AI art 9 years later. As of now, there has been no statement from the studio, and we do not know how he feels about all of this. He most likely hates AI companies stealing his style but does he also hate fans for generating personal photos because they like his style? out of context in 2016 Here’s the full video. Here’s the full video. Here’s the full video. the full video Shouldn’t we blame and shame people for using this? As always, the anger is more directed towards regular people by those who feel intellectually superior. It is easy to punch below the belt. So, instead of protesting against our overlords, people find it much easier to criticize and hate the common people. The problem isn’t regular people generating cute pics for their own amusement; the problem is corporations stealing the work of everyone to train AI and putting people out of their jobs. The problem is when you use those works (like Studio Ghibli, which always promoted peace) to create hate speech and political propaganda. And, no common folk is using the Ghibli style and claiming it as their own original style. It’s an inspiration and a homage. If you think it’s wrong, would you have also said the same thing if an artist hand-drew something in a popular style as a homage? Is it an insult to Michael Jackson to do the moonwalk poorly? Is it an insult to the original singer if I sing a song poorly in front of my friends? Is AI-generated art not art? What if it’s based on my unique input? Many people would argue that a banana taped to a wall is not art, but people have paid millions for things like that. So, art can’t be defined by boundaries. If a digital artist has a character in mind and uses existing vectors and elements provided by the sketching tool to create their work faster, is that not art? If it is art, why is it not art if somebody is bringing their imagination to life using AI? Both have the same outcome, with only the technique being different. Yes, the art style is a remix of what has come before, but do all hand artists use original art styles without doing any derivative work? If you say AI art is still art, why are you saying art dies today? Art will always exist in some form. What dies today is the birth of groundbreaking art styles because: It is simply no longer a good career option for most to pursue.
Businesses wouldn’t invest resources into creating any genre-defining work (the next Miyazaki) because it isn’t worth it. When good enough can be created at 1/100th the cost, most businesses would avoid investing in traditional styles. It is simply no longer a good career option for most to pursue. Businesses wouldn’t invest resources into creating any genre-defining work (the next Miyazaki) because it isn’t worth it. When good enough can be created at 1/100th the cost, most businesses would avoid investing in traditional styles. “Photography did not kill painting and digital art didn’t kill traditional art”? “Photography did not kill painting and digital art didn’t kill traditional art”? Yes, photography did not kill painting, but is the demand for paintings the same now as it was during the French Revolution? Also, digital art did not eliminate traditional art; it significantly devalued it. Would people pay 10X more for human digital art when AI art can do the job? Forget people, will businesses pay? Will art become a more premium product? Yes, scarcity drives value, but that will be reserved for only the best. Even though handloomed clothes are now expensive, most handloom workers in India are displaced. The same will happen to art and other fields. Unlike in the past, when high-quality work commanded top dollar while mediocre work still earned a livable wage, the future where AI surpasses mediocrity will leave room only for the very best to sustain a career in many fields. That also means those who imitate will no longer have a career because AI does all the imitation now! Can we stop this? No, there is no putting this genie back in the bottle. If a country regulates or restricts AI, some other country will make it. And, even if a scenario existed, open-sourced copies of several tools exist which can be run locally by businesses. Plus, do remember that businesses control the governments in most parts of the world. If they want AI, they will use AI. It might be unfortunate but there is no other way!

Wholeheartedly Embracing the Studio Ghibli AI Trend as Art Dies Today!

Celé srdce přijímá Studio Ghibli AI Trend jako umění umírá dnes!

¡Adoptemos con todo el corazón la tendencia de inteligencia artificial de Studio Ghibli mientras el arte muere hoy!

今日、芸術が消滅する中、スタジオジブリの AI トレンドを心から受け入れます!

Pilnīgi pieņemot Studio Ghibli AI tendenci kā māksla mirst šodien!

Целосно прифаќање на студиото Ghibli AI Trend како уметност умира денес!

Studio Ghibli-ийн AI Trend-ийг бүгдийгөө олж авах Art Dies Today!

Dengan Sepenuh Hati Memeluk Trend AI Studio Ghibli sebagai Seni Mati Hari Ini!

په بشپړه توګه د سټرایډ Ghibli AI ټینډ راټول کړئ لکه څنګه چې د آرټ آج مرې!

Z celého srdca prijímame trend umelej inteligencie Studio Ghibli, keďže umenie dnes zomiera!

Kumbundira Nemoyo wese Studio Ghibli AI Trend sezvo Art Inofa Nhasi!

Si Kalsooni Lahayn ah Isbeddelka Studio Ghibli AI sida Fanku u dhinto Maanta!

A Good Story Is the Difference Between Success and Survival

Dobrý příběh je rozdíl mezi úspěchem a přežitím

Una buena historia es la diferencia entre el éxito y la supervivencia

Une bonne histoire fait la différence entre le succès et la survie

एक अच्छी कहानी सफलता और अस्तित्व के बीच का अंतर बताती है

Dobra priča je razlika između uspjeha i preživljavanja

Լավ պատմություն, որ տարբերություն է հաջողության եւ կյանքի միջեւ

良い物語は成功と生存の違いです。

ERROR: invalid target_lang (ky)

یو ښه داستان د بریالیتوب او د ژوند په منځ کې فرق دی

O poveste bună este diferența dintre succes și supraviețuire

ERROR: invalid target_lang (tg)

Stories, News & Mentions about CLEARLY

Clearly’s Stories on HackerNoon