Semantic Equivalence of e-Commerce Queries

•

0 likes•1,481 views

Title: Semantic Equivalence of e-Commerce Queries Authors: Aritra Mandal, Daniel Tunkelang, Zhe Wu Presented at KDD 2023 Workshop on E-Commerce and Natural Language Processing (ECNLP 2023).

Software

Semantic Equivalence of
e-Commerce Queries
Aritra Mandal and Daniel Tunkelang and Zhe Wu
eBay Inc.

Search query != search intent.
● Information retrieval researchers worry about queries that map to multiple intents.
jaguar or ?
● Practitioners worry more about multiple queries that map to the same intent.
lightning to 3.5mm
iphone to aux

Equivalent queries should yield equivalent experiences.
Recall?
CTR?
Conversion Rate?
...
?
or

Opportunity to increase recall while preserving precision.
Similar but not
equivalent intent.

High-level strategy to leverage query equivalence.
Map queries to vectors.
Store in nearest-neighbor database.
(i.e., optimize for user
or business outcome)

Two strategies for recognizing equivalent queries.
● Surface Similarity
○ Variation in inflection, word order, compounding, noise words.
black tshirts for men = mens black t-shirt =
● Behavioral Similarity
○ Queries lead to engagement with equivalent or similar results.
lightning to 3.5mm = iphone to aux =

Introducing the “bag of documents” model.

Query vectors are centroids of associated product vectors
►
►
[0.13, 0.81, … ]
[0.09, 0.75, … ]
…
►
[0.11, 0.79, … ]
[0.13, 0.81, … ]
[0.09, 0.77, … ]
…
►
[0.12, 0.78, … ]
►
cos > 0.98
black tshirts for men mens black t-shirt

$Works well, but only for head and torso queries. ● Offline approach works for queries with enough engagement history. ● Would be expensive to compute aggregates of result vectors online. ● Still, head and torso queries tend to represent a large fraction of traffic.$

Train online sentence transformer model for tail queries.
● Train using (query1, query2, similarity) triples from offline model.
● Oversample similar query pairs to increase sensitivity where it matters.
● Fine-tune a pre-trained micro-BERT sentence transformer model.
● Concatenate the output of a query classifier to the query keywords.

Architecture for Online Query Similarity Model

Results
Model Dataset Name Pearson’s correlation
query-sim-ecom eBay Internal 0.87
query-sim-ecom ESCI query-query 0.85
all-MiniLM-L12-v2 ESCI query-query 0.68
Query 1 Query 2 cosine
hdmi to galaxy s8 s9 hdmi 0.9993
movie money prop money 0.9995
cassette adapter for iphone tape to aux 0.9993
Examples from ESCI
of queries with low
surface but high
behavioral similarity:

Summary
● Queries with equivalent intent should yield equivalent experiences.
● Query similarity can increase recall while preserving precision.
● Signals can come from either surface or behavioral similarity.
● Offline bag-of-documents model: queries as means of product vectors.
● Fine-tune online Micro-BERT sentence transformer model for tail queries.
● It just works!

Similar to Semantic Equivalence of e-Commerce Queries

Comparable entity mining from comparative questions

IEEEFINALYEARPROJECTS

The talk "Strategies for using alternative queries to mitigate zero results and their application to online marketplaces" took center stage at Haystack EU 2023. The presenters: Jean Silva @wallapop, ex @trivago Working with various programming languages since 2008 and fall in love with the search world in 2014. Jean has worked in different industries such as travel, e-commerce, and classified listings. René Kriegler @OpenSourceConnections Works in search since ~2008 Jean and René shed light on the challenges associated with zero results queries, emphasizing the critical importance of avoiding empty search result lists to enhance user experience and prevent negative business impacts. This issue is particularly pertinent in user-generated content platforms like Wallapop, where the diversity of items and user backgrounds complicates search outcomes. Addressing these challenges, they discussed recent advancements in vector search and the availability of powerful language models for semantic search. While these developments offer potential solutions by ensuring some results are always returned, they also pose usability and computational challenges, especially in platforms with short-lived content. Building upon strategies presented at Haystack US 2019, Jean and René explored approaches to providing users with alternative queries, such as query relaxation and term substitution, leveraging Large Language Models. Additionally, they shared insights from experiments involving simpler query relaxation techniques based on query term statistics. These strategies not only improve user experience but also offer versatile alternatives to vector search, applicable beyond classified ads platforms. For those interested in delving deeper into these discussions, the conference talk can be accessed via the following link: https://haystackconf.com/eu2023/talk-9/ Jean Silva Search Engineer @Wallapop.

Strategies for using alternative queries to mitigate zero results

Jean Silva

Introduction to Machine Learning

SATHVIK MANIKANTAN N U

Sentiment Analysis: A comparative study of Deep Learning and Machine Learning

IRJET Journal

Scott is co-founder and CEO of SigOpt, a YC and a16z backed “Optimization as a Service” startup in San Francisco. Scott has been applying optimal learning techniques in industry and academia for years, from bioinformatics to production advertising systems. Before SigOpt, Scott worked on the Ad Targeting team at Yelp leading the charge on academic research and outreach with projects like the Yelp Dataset Challenge and open sourcing MOE. Scott holds a PhD in Applied Mathematics and an MS in Computer Science from Cornell University and BS degrees in Mathematics, Physics, and Computational Physics from Oregon State University. Scott was chosen as one of Forbes’ 30 under 30 in 2016. Abstract summary Bayesian Global Optimization: Using Optimal Learning to Deep Learning Models: In this talk we introduce Bayesian Optimization as an efficient way to optimize machine learning model parameters, especially when evaluating different configurations is time-consuming or expensive. Deep learning pipelines are notoriously expensive to train and often have many tunable parameters including hyperparameters, the architecture, feature transformations that can have a large impact on the efficacy of the model. We will motivate the problem by giving several example applications using open source deep learning frameworks and open datasets. We’ll compare the results of Bayesian Optimization to standard techniques like grid search, random search, and expert tuning.

Scott Clark, CEO, SigOpt, at MLconf Seattle 2017

MLconf

In this talk we introduce Bayesian Optimization as an efficient way to optimize machine learning model parameters, especially when evaluating different parameters is time-consuming or expensive. Deep learning pipelines are notoriously expensive to train and often have many tunable parameters including hyperparameters, the architecture, feature transformations that can have a large impact on the efficacy of the model. We will motivate the problem by giving several example applications using multiple open source deep learning frameworks and open datasets. We’ll compare the results of Bayesian Optimization to standard techniques like grid search, random search, and expert tuning.

MLconf 2017 Seattle Lunch Talk - Using Optimal Learning to tune Deep Learning...

SigOpt

IOSR Journal of Computer Engineering (IOSR-JCE) is a double blind peer reviewed International Journal that provides rapid publication (within a month) of articles in all areas of computer engineering and its applications. The journal welcomes publications of high quality papers on theoretical developments and practical applications in computer technology. Original research papers, state-of-the-art reviews, and high quality technical notes are invited for publications.

Extraction of Data Using Comparable Entity Mining

iosrjce

E017252831

IOSR Journals

Demystifying Machine Learning

Ayodele Odubela

Behavioral economics transformed how we think about human decision making, rejecting expected utility maximization for the real world of heuristics, biases, and satisficing. In this talk, I'll argue that our thinking about search engines needs a similar transformation. I will compare the Probability Ranking Principle to expected utility maximization and offer ways that AI can help searchers satisfice through query understanding. This was an invited talk given at the 2023 Walmart AI Summit. Speaker Bio Daniel Tunkelang is an independent consultant specializing in search, machine learning / AI, and data science. He completed undergraduate and master's degrees in Computer Science and Math at MIT and a PhD in computer science at CMU. He was a founding employee and chief scientist of Endeca, a search pioneer that Oracle acquired in 2011. He then led engineering and data science teams at Google and LinkedIn. He has written a book on Faceted Search, and he blogs on Medium about search-related topics — particularly query understanding. He has worked with numerous tech companies, retailers, and others, including Algolia, Apple, Canva, Coupang, eBay, Etsy, Flipkart, Home Depot, Oracle, Pinterest, Salesforce, Target, Yelp, and Zoom.

Helping Searchers Satisfice through Query Understanding

Daniel Tunkelang

How to fine-tune and develop your own large language model.pptx

Knoldus Inc.

Designing cloud solutions can be a challenging task. Complications occur when critical specification criteria are unavailable. Crowd-sourced data provides knowledge and understanding, enabling the calculation of any missing criteria. Analytical intelligence, machine learning and cloud community data form the recipe for successfully drafting cloud solutions. View the VMworld 2017 vBrownBag Tech Talk on youtube: https://youtu.be/DqojNKVPQKE

VMworld vBrownbag vmtn6739e - machine learning (ai) for workload analytics an...

Kenneth Moore

Movie Recommendation System.pptx

randominfo

Neel Sundaresan - Teaching a machine to code

MLconf

ChatGPT-and-Generative-AI-Landscape Working of generative ai search

rohitcse52

This talk was recorded in London on Oct 30, 2018 and can be viewed here: https://youtu.be/p4iAnxwC_Eg The good news is building fair, accountable, and transparent machine learning systems is possible. The bad news is it’s harder than many blogs and software package docs would have you believe. The truth is nearly all interpretable machine learning techniques generate approximate explanations, that the fields of eXplainable AI (XAI) and Fairness, Accountability, and Transparency in Machine Learning (FAT/ML) are very new, and that few best practices have been widely agreed upon. This combination can lead to some ugly outcomes! This talk aims to make your interpretable machine learning project a success by describing fundamental technical challenges you will face in building an interpretable machine learning system, defining the real-world value proposition of approximate explanations for exact models, and then outlining the following viable techniques for debugging, explaining, and testing machine learning models Mateusz is a software developer who loves all things distributed, machine learning and hates buzzwords. His favourite hobby data juggling. He obtained his M.Sc. in Computer Science from AGH UST in Krakow, Poland, during which he did an exchange at L’ECE Paris in France and worked on distributed flight booking systems. After graduation he move to Tokyo to work as a researcher at Fujitsu Laboratories on machine learning and NLP projects, where he is still currently based.

Machine Learning Interpretability - Mateusz Dymczyk - H2O AI World London 2018

Sri Ambati

Usage of AI and machine learning models is likely to become more commonplace as larger swaths of the economy embrace automation and data-driven decision-making. While these predictive systems can be quite accurate, they have been treated as inscrutable black boxes in the past, that produce only numeric predictions with no accompanying explanations. Unfortunately, recent studies and recent events have drawn attention to mathematical and sociological flaws in prominent weak AI and ML systems, but practitioners usually don’t have the right tools to pry open machine learning black-boxes and debug them. This presentation introduces several new approaches to that increase transparency, accountability, and trustworthiness in machine learning models. If you are a data scientist or analyst and you want to explain a machine learning model to your customers or managers (or if you have concerns about documentation, validation, or regulatory requirements), then this presentation is for you!

Interpretable Machine Learning

Sri Ambati

In this workshop we covered an introduction to Generative AI and Large Language Models (LLMs), an explanation of AWS Foundation Models and their role in providing pre-trained LLMs, the benefits of leveraging LLMs in enterprises, deploying LLMs on AWS Infrastructure including infrastructure requirements and available AWS services and tools, and a demo showcasing Text-to-Image and Text Summarization using Foundation Models, as well as utilising Retrieval Augmented Generation and LangChain with AWS tools for Enterprise use cases. Connect with me for interesting session in future @https://www.linkedin.com/in/jayyanar/

AWS_Meetup_BLR_July_22_Social.pdf

Ayyanar Jeyakrishnan

Deep learning Introduction and Basics

Nitin Mishra

Everything you need to know about AutoML

Arpitha Gurumurthy

Similar to Semantic Equivalence of e-Commerce Queries (20)

Comparable entity mining from comparative questions

Strategies for using alternative queries to mitigate zero results

Introduction to Machine Learning

Sentiment Analysis: A comparative study of Deep Learning and Machine Learning

Scott Clark, CEO, SigOpt, at MLconf Seattle 2017

MLconf 2017 Seattle Lunch Talk - Using Optimal Learning to tune Deep Learning...

Extraction of Data Using Comparable Entity Mining

E017252831

Demystifying Machine Learning

Helping Searchers Satisfice through Query Understanding

How to fine-tune and develop your own large language model.pptx

VMworld vBrownbag vmtn6739e - machine learning (ai) for workload analytics an...

Movie Recommendation System.pptx

Neel Sundaresan - Teaching a machine to code

ChatGPT-and-Generative-AI-Landscape Working of generative ai search

Machine Learning Interpretability - Mateusz Dymczyk - H2O AI World London 2018

Interpretable Machine Learning

AWS_Meetup_BLR_July_22_Social.pdf

Deep learning Introduction and Basics

Everything you need to know about AutoML

More from Daniel Tunkelang

MMM, Search! An opinionated discussion of search metrics, models, and methods. Presented to the Wikimedia Foundation on April 27, 2020. About the Speaker Daniel Tunkelang is an independent consultant specializing in search, discovery, machine learning / AI, and data science. He was a founding employee of Endeca, a search pioneer that Oracle acquired. After 10 years at Endeca, he moved to Google, where he led a local search team. He then served as a director of data science and search at LinkedIn. After leaving LinkedIn in 2015, he became an independent consultant. His clients have included Apple, eBay, Coupang, Etsy, Flipkart, Gartner, Pinterest, Salesforce, and Yelp; as well as some of the largest traditional retailers. Daniel completed undergraduate and master's degrees in Computer Science and Math at MIT and a Ph.D. in computer science at CMU. He wrote a book on Faceted Search, published by Morgan & Claypool, and he blogs on Medium about search-related topics -- particularly about query understanding. He is also active on Twitter, LinkedIn, and Quora.

MMM, Search!

Daniel Tunkelang

Enterprise Intelligence: Putting the Pieces Together http://enterpriserelevance.com/kdd2016/keynote.html These slides are for a keynote presentation delivered at the Workshop on Enterprise Intelligence, held in conjunction with the 22nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2016). About the author: Daniel Tunkelang is a data science and engineering executive who has built and led some of the strongest teams in the software industry. He studied computer science and math at MIT and has a PhD in computer science from CMU. He was a founding employee and chief scientist of Endeca, a search pioneer that Oracle acquired for $1.1B. He led a local search team at Google. He was a director of data science and engineering at LinkedIn, and he established their query understanding team. Daniel is a widely recognized writer and speaker. He is frequently invited to speak at academic and industry conferences, particularly in the areas of information retrieval, web science, and data science. He has written the definitive textbook on faceted search (now a standard for ecommerce sites), established an annual symposium on human-computer interaction and information retrieval, and authored 24 US patents. His social media posts have attracted over a million page views. Daniel advises and consults for companies that can benefit strategically from his expertise. His clients range from early-stage startups to "unicorn" technology companies like Etsy and Pinterest. He helps companies make decisions around algorithms, technology, product strategy, hiring, and organizational structure.

Enterprise Intelligence

Daniel Tunkelang

Query Understanding: A Manifesto

Daniel Tunkelang

Where should you put your data scientists?

Daniel Tunkelang

Data Science: A Mindset for Productivity Keynote at 2015 Ronin Labs West Coast CTO Summit https://www.eventjoy.com/e/west-coast-cto-summit-2015 Abstract Data science isn't just about using a collection of technologies and algorithms. Data science requires a mindset that solves problems at a higher level of abstraction. How do we model utility when we think about optimization? How do we decide which hypotheses to test? How do we allocate our scarce resources to make progress? There are no silver bullets. But I'll share what I've learned from a variety of contexts over the course of my work at Endeca, Google, and LinkedIn; and I hope you'll leave this talk with some practical wisdom you can apply to your next data science project.

Data Science: A Mindset for Productivity

Daniel Tunkelang

My Three Ex’s: A Data Science Approach for Applied Machine Learning Daniel Tunkelang (LinkedIn) Presented at QCon San Francisco 2014 in the Applied Machine Learning and Data Science track https://qconsf.com/presentation/my-three-ex%E2%80%99s-data-science-approach-applied-machine-learning Abstract This talk is about applying machine learning to solve problems. It’s not a talk about machine learning — or at least not about the theory of machine learning. Theoretical machine learning requires a deep understanding of computer science and statistics. It’s one of the most studied areas of computer science, and advances in theoretical machine learning give us hope of solving the world’s “AI-hard” problems. Applied machine learning is more grounded but no less important. We are surrounded by opportunities to apply classifiers, learn rules, compute similarity, and assemble clusters. We don’t need to develop new algorithms for any of these problems — our textbooks and open-source libraries have done that hard work for us. But algorithms are not enough. Applying machine learning to solve problems requires a data science mindset that transcends the algorithmic details. In this talk, I’ll communicate the data science mindset by describing my three ex’s: express, explain, and experiment. These three activities are the pillars of a successful strategy for applying machine learning to solve problems. Whether you’re a machine learning novice or expert, I hope you’ll leave this talk with some practical wisdom you can apply to your next project.

My Three Ex’s: A Data Science Approach for Applied Machine Learning

Daniel Tunkelang

Web Science: How is it different? Daniel Tunkelang, LinkedIn Keynote Address at ACM Web Science 2014 Conference The scientific method of observation, measurement, and experiment may be our greatest achievement as a species. The technological innovation we enjoy today is the product of a culture of systematized scientific experimentation. But historically scientific experimentation has been expensive. Experiments consumed natural resources, took a long time to conduct, and required even more time and labor to analyze. In order to be productive, scientists have had to factor these costs into their work and to optimize accordingly. Web science is different. Not, as some have speciously argued, because big data has made the scientific method obsolete. The key difference is that web science has changed the economics of scientific experimentation. Thus, even as web scientists apply the traditional scientific method, they optimize based on very different economics. In this talk, I'll survey how web science has changed our approach to experimentation, for better and for worse. Specifically, I'll talk about differences in hypothesis generation, offline analysis, and online testing. Bio Daniel Tunkelang is Head of Query Understanding at LinkedIn, where he previously formed and led the product data science team. LinkedIn search allows members to find people, companies, jobs, groups and other content. His team aims to provide users with the best possible results that satisfy their information needs and help to get insights from professional data. Tunkelang has BS and MS degrees in computer science and math from MIT, and a PhD in computer science from CMU. He co-founded the annual symposium on human-computer interaction and information retrieval (HCIR) and wrote the first book on Faceted Search (Morgan and Claypool 2009). Prior to joining LinkedIn, Tunkelang was Chief Scientist of Endeca (acquired by Oracle in 2011 for $1.1B) and leader of the local search quality team at Google, mapping local businesses to their home pages. He is the co-inventor of 20 patents.

Web science - How is it different?

Daniel Tunkelang

Better Search Through Query Understanding Presented as a Data Talk at Intuit on April 22, 2014 Search is a fundamental problem of our time — we use search engines daily to satisfy a variety of personal and professional information needs. But search engine development still feels stuck in an information retrieval paradigm that focuses on result ranking. In this talk, I’ll advocate an emphasis on query understanding. I’ll talk about how we implement query understanding at LinkedIn, and I’ll present examples from the broader web. Hopefully you’ll come out with a different perspective on search and share my appreciation for how we can improve search through query understanding. About the Speaker Daniel Tunkelang leads LinkedIn's efforts around query understanding. Before that, he led LinkedIn's product data science team. He previously led a local search quality team at Google and was a founding employee of Endeca (acquired by Oracle in 2011). He has written a textbook on faceted search, and is a recognized advocate of human-computer interaction and information retrieval (HCIR). He has a PhD in Computer Science from CMU, as well as BS and MS degrees from MIT.

Better Search Through Query Understanding

Daniel Tunkelang

Keynote at CIKM 2013 Workshop on Data-driven User Behavioral Modelling and Mining from Social Media Social Search in a Professional Context Daniel Tunkelang (LinkedIn) Social networks bring a new dimension to search. Instead of looking for web pages or text documents, LinkedIn members search a world of entities connected by a rich graph of relationships. Search is a fundamental part of the LinkedIn ecosystem, as it helps our members find and be found. Unlike most search applications, LinkedIn's search experience is highly personalized: two LinkedIn members performing the same search query are likely to see completely different results. Delivering the right results to the right person depends on our ability to leverage our each member's unique professional identity and network. In this talk, I'll describe the kinds of search behavior we see on LinkedIn, and some of the approaches we've taken to help our members address their information needs.

Social Search in a Professional Context

Daniel Tunkelang

Find and Be Found: Information Retrieval at LinkedIn SIGIR 2013 Industry Track Presentation http://sigir2013.ie/industry_track.html LinkedIn has a unique data collection: the 200M+ members who use LinkedIn are also the most valuable entities in our corpus, which consists of people, companies, jobs, and a rich content ecosystem. Our members use LinkedIn to satisfy a diverse set of navigational and exploratory information needs, which we address by leveraging semi-structured and social content to understanding their query intent and deliver a personalized search experience. In this talk, we will discuss some of the unique challenges we face in building the LinkedIn search platform, the solutions we've developed so far, and the open problems we see ahead of us. Shakti Sinha heads LinkedIn's search relevance team, and has been making key contributions to LinkedIn's search products since 2010. He previously worked at Google as both a research intern and a software engineer. He has an MS in Computer Science from Stanford, as well as a BS degree from College of Engineering, Pune. Daniel Tunkelang leads LinkedIn's efforts around query understanding. Before that, he led LinkedIn's product data science team. He previously led a local search quality team at Google and was a founding employee of Endeca (acquired by Oracle in 2011). He has written a textbook on faceted search, and is a recognized advocate of human-computer interaction and information retrieval (HCIR). He has a PhD in Computer Science from CMU, as well as BS and MS degrees from MIT.

Find and be Found: Information Retrieval at LinkedIn

Daniel Tunkelang

Search as Communication: Lessons from a Personal Journey by Daniel Tunkelang (Head of Query Understanding, LinkedIn) Presented at Etsy's Code as Craft Series on May 21, 2013 When I tell people I spent a decade studying computer science at MIT and CMU, most assume that I focused my studies in information retrieval — after all, I’ve spent most of my professional life working on search. But that’s not how it happened. I learned about information extraction as a summer intern at IBM Research, where I worked on visual query reformulation. I learned how search engines work by building one at Endeca. It was only after I’d hacked my way through the problem for a few years that I started to catch up on the rich scholarly literature of the past few decades. As a result, I developed a point of view about search without the benefit of academic conventional wisdom. Specifically, I came to see search not so much as a ranking problem as a communication problem. In this talk, I’ll explain my communication-centric view of search, offering examples, general techniques, and open problems. -- Daniel Tunkelang is Head of Query Understanding at LinkedIn. Educated at MIT and CMU, he has his career working on big data, addressing key challenges in search, data mining, user interfaces, and network analysis. He co-founded enterprise search and business intelligence pioneer Endeca, where he spent a decade as its Chief Scientist. In 2011, Endeca was acquired by Oracle for over $1B. Previous to LinkedIn, he led a team at Google working on local search quality. Daniel has authored fifteen patents, written a textbook on faceted search, and created the annual symposium on human-computer interaction and information retrieval.

Search as Communication: Lessons from a Personal Journey

Daniel Tunkelang

Enterprise Search: How Do We Get There From Here? by Daniel Tunkelang (Head of Query Understanding, LinkedIn) Keynote at 2013 Enterprise Search Summit We've been tackling the challenges of enterprise and site search for at least 3 decades. We've succeeded to the point that search is the gateway to many of our information repositories. Nonetheless, users of enterprise search systems are frustrated with these systems' shortcomings. We see this frustration in surveys, but, more importantly, most of us experience it personally in our daily work life. We all dream of a world where searching any information repository is as effective as searching the web—perhaps even more so. A world where we find what we're looking for, or quickly determine that it doesn't exist. Is this Utopia possible? If so, how do we get there from here? Or at least somewhere close? In this talk, Tunkelang reviews the track record of enterprise search. He talks about what's worked and what hasn't, especially as compared to web search. Finally, he proposes some paths to bring us closer to our dream. -- Daniel Tunkelang is Head of Query Understanding at LinkedIn. Educated at MIT and CMU, he has his career working on big data, addressing key challenges in search, data mining, user interfaces, and network analysis. He co-founded enterprise search and business intelligence pioneer Endeca, where he spent a decade as its Chief Scientist. In 2011, Endeca was acquired by Oracle for over $1B. Previous to LinkedIn, he led a team at Google working on local search quality. Daniel has authored fifteen patents, written a textbook on faceted search, and created the annual symposium on human-computer interaction and information retrieval.

Enterprise Search: How do we get there from here?

Daniel Tunkelang

Big Data, We Have a Communication Problem by Daniel Tunkelang Presented on April 30, 2013 at the TTI/Vanguard Conference on Ginormous Systems http://www.ttivanguard.com/conference/2013/ginormous.html It's a cliché that we live in a world of Big Data. But the bottleneck in understanding data is not computational. Rather, the biggest challenge is designing technical solutions that effectively leverage human cognitive ability. Data analysis systems should augment people's capabilities rather than replace them. This argument is as old as computer science itself: in 1962, Doug Engelbart said that the goal of technology is “the enhancement of human intellect by increasing the capability of a human to approach a complex problem situation.” Algorithms extract signal from raw data, but people fill in the gaps, creating models and evaluating analyses. Empowering people to understand data is not just a surface problem of building better interfaces and visualizations. We need to interact with data not only after performing computational analysis, but throughout the analysis process in order to improve our models and algorithms. In order to do so, we need tools and processes specifically designed to offer people transparency, guidance, and control. Human-computer information retrieval has been revolutionizing our approach to information seeking -- no modern search engine limits users to black-box relevance ranking and ten blue links. We need to take similar steps in our analysis of big data, making people the center of the analysis process and developing the technical innovations that enable people to fulfill this role.

Big Data, We Have a Communication Problem

Daniel Tunkelang

How To Interview a Data Scientist Daniel Tunkelang Presented at the O'Reilly Strata 2013 Conference Video: https://www.youtube.com/watch?v=gUTuESHKbXI Interviewing data scientists is hard. The tech press sporadically publishes “best” interview questions that are cringe-worthy. At LinkedIn, we put a heavy emphasis on the ability to think through the problems we work on. For example, if someone claims expertise in machine learning, we ask them to apply it to one of our recommendation problems. And, when we test coding and algorithmic problem solving, we do it with real problems that we’ve faced in the course of our day jobs. In general, we try as hard as possible to make the interview process representative of actual work. In this session, I’ll offer general principles and concrete examples of how to interview data scientists. I’ll also touch on the challenges of sourcing and closing top candidates.

How to Interview a Data Scientist

Daniel Tunkelang

Information, Attention, and Trust: A Hierarchy of Needs

Daniel Tunkelang

Data By The People, For The People Daniel Tunkelang Director, Data Science at LinkedIn Invited Talk at the 21st ACM International Conference on Information and Knowledge Management (CIKM 2012) LinkedIn has a unique data collection: the 175M+ members who use LinkedIn are also the content those same members access using our information retrieval products. LinkedIn members performed over 4 billion professionally-oriented searches in 2011, most of those to find and discover other people. Every LinkedIn search and recommendation is deeply personalized, reflecting the user's current employment, career history, and professional network. In this talk, I will describe some of the challenges and opportunities that arise from working with this unique corpus. I will discuss work we are doing in the areas of relevance, recommendation, and reputation, as well as the ecosystem we have developed to incent people to provide the high-quality semi-structured profiles that make LinkedIn so useful. Bio: Daniel Tunkelang leads the data science team at LinkedIn, which analyzes terabytes of data to produce products and insights that serve LinkedIn's members. Prior to LinkedIn, Daniel led a local search quality team at Google. Daniel was a founding employee of faceted search pioneer Endeca (recently acquired by Oracle), where he spent ten years as Chief Scientist. He has authored fourteen patents, written a textbook on faceted search, created the annual workshop on human-computer interaction and information retrieval (HCIR), and participated in the premier research conferences on information retrieval, knowledge management, databases, and data mining (SIGIR, CIKM, SIGMOD, SIAM Data Mining). Daniel holds a PhD in Computer Science from CMU, as well as BS and MS degrees from MIT.

Data By The People, For The People

Daniel Tunkelang

Content, Connections, and Context Daniel Tunkelang, LinkedIn Keynote at Workshop on Recommender Systems and the Social Web At 6th ACM International Conference on Recommender Systems (RecSys 2012) Recommender systems for the social web combine three kinds of signals to relate the subject and object of recommendations: content, connections, and context. Content comes first - we need to understand what we are recommending and to whom we are recommending it in order to decide whether the recommendation is relevant. Connections supply a social dimension, both as inputs to improve relevance and as social proof to explain the recommendations. Finally, context determines where and when a recommendation is appropriate. I'll talk about how we use these three kinds of signals in LinkedIn's recommender systems, as well as the challenges we see in delivering social recommendations and measuring their relevance.

Content, Connections, and Context

Daniel Tunkelang

Keynote at 2012 Semantic Technology and Business Conference Scale, Structure, and Semantics Daniel Tunkelang, LinkedIn Science fiction has a mixed track record when it comes to anticipating technological innovations. While Jules Verne fared well with with his predictions of submarine and space technology, artificial intelligence hasn't produced anything like Arthur C. Clarke's HAL 9000. Instead, we've managed to elicit intelligence from machines through unexpected means. Search engines have achieved remarkable success in organizing the world's information by crawling the web, indexing documents, and exploiting link structure to establish authoritativeness. At LinkedIn, we apply large-scale analytics to terabytes of semistructured data to deliver products and insights that serve our 150M+ members. Semantics emerge when we apply the right analytical techniques to a sufficient quality and quantity of data. In this talk, I will describe how LinkedIn's huge and rich graph of relationship data that powers the products our users love. I believe that the lessons we have learned apply broadly to other semantic applications. While quantity and quality of data are the key challenges to delivering a semantically rich experience, the key is to create the right ecosystem that incents people to give you good data, which then forms the basis for great data products.

Scale, Structure, and Semantics

Daniel Tunkelang

Presentation from O'Reilly Strata 2012 on Big Data Humans, Machines, and the Dimensions of Microwork Daniel Tunkelang (LinkedIn) Claire Hunsaker (Samasource) The advent of crowdsourcing has wildly expanded the ways we think of incorporating human judgments into computational workflows. Computer scientists, economists, and sociologists have explored how to effectively and efficiently distribute microwork tasks to crowds and use their work as inputs to create or improve data products. Simultaneously, crowdsourcing providers are exploring the bounds of mechanical QA flows, worker interfaces, and workforce management systems. But what tasks should be performed by humans rather than algorithms? And what makes a set of human judgments robust? Quantity? Consensus? Quality or trustworthiness of the workers? Moreover, the robustness of judgments depends not only on the workers, but on the task design. Effective crowdsourcing is a cooperative endeavor. In this talk, we will analyze various dimensions of microwork that characterize applications, tasks, and crowds. Drawing on our experience at companies that have pioneered the use of microwork (Samasource) and data science (LinkedIn), we will offer practical advice to help you design crowdsourcing workflows to meet your data product needs.

Strata 2012: Humans, Machines, and the Dimensions of Microwork

Daniel Tunkelang

These slides are from a tutorial at the 5th ACM International Conference on Recommender Systems (RecSys 2011). Recommender systems aim to provide users with products or content that satisfy the users' stated or inferred needs. The primary evaluation measures for recommender systems emphasize either the perceived relevance of the recommendations or the actions associated with those recommendations (e.g., purchases or clicks). Unfortunately, this transactional emphasis neglects how users interact with recommendations in the context of information seeking tasks. The effectiveness of this interaction determines the user's experience beyond a single transaction. This tutorial explores the role of recommendations as part of a conversation between the user and an information seeking system. The tutorial does not require any special background in interfaces or usability, and will focus on practical techniques to make recommender systems most effective for users.

Recommendations as a Conversation with the User

Daniel Tunkelang

More from Daniel Tunkelang (20)

MMM, Search!

Enterprise Intelligence

Query Understanding: A Manifesto

Where should you put your data scientists?

Data Science: A Mindset for Productivity

My Three Ex’s: A Data Science Approach for Applied Machine Learning

Web science - How is it different?

Better Search Through Query Understanding

Social Search in a Professional Context

Find and be Found: Information Retrieval at LinkedIn

Search as Communication: Lessons from a Personal Journey

Enterprise Search: How do we get there from here?

Big Data, We Have a Communication Problem

How to Interview a Data Scientist

Information, Attention, and Trust: A Hierarchy of Needs

Data By The People, For The People

Content, Connections, and Context

Scale, Structure, and Semantics

Strata 2012: Humans, Machines, and the Dimensions of Microwork

Recommendations as a Conversation with the User

Recently uploaded

Craft an AI & Machine Learning Pitch with our Editable Professional PowerPoint Template. Ignite your AI & Machine Learning pitch with our cutting-edge PowerPoint template tailored for the industry. Perfect for AI conferences, investor presentations, sales pitches to tech-focused companies, training sessions, and educational programs. - 20+ editable slides: Get a variety of options to choose from for your presentation. - Time-saving solution: Download, replace text/images with a few clicks. - User-friendly customization: Easy to use and personalize. - Modern and attractive design: Captivating visuals, sleek layout. - Tailored to your requirements: Fully alterable for customization. - Well-organized slides: Complete control over content. - Thematic specificity: Reflects healthcare industry with relevant graphics. - Showcase your business idea: Communicate value proposition effectively.

AI & Machine Learning Presentation Template

Presentation.STUDIO

Data spaces in distributed environments should be allowed to evolve in agile ways providing data space owners with large flexibility about which data they store. Agility and heterogeneity, however, jeopardize data exchanges because representations may build on varying ontologies and data consumers may not rely on the semantic correctness of their queries in the context of semantically heterogeneous, evolving data spaces. Graph data spaces are one example of a powerful model for representing and querying data whose semantics may change over time. To assert and enforce conditions on individual graph data spaces, shape languages (e.g SHACL) have been developed. We investigate the question of how querying and programming can be guarded by reasoning over SHACL constraints in a distributed setting and we sketch a picture of how a future landscape based on semantically heterogeneous data spaces might look like.

Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...

Steffen Staab

Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...

SelfMade bd

SHRMPro HRMS Software Solutions Presentation

Shrmpro

At TECUNIQUE, we're a stable and steadily growing Indian software services company with over 14 years of industry experience. Specializing in offshore software development and quality assurance services, we've built a reputation for delivering unique and effective solutions to start-ups, software development companies, enterprises, and digital agencies. We pride ourselves on our commitment to excellence and innovation. By blending insightful business domain knowledge with exceptional technical prowess, we craft tailor-made solutions that meet the unique needs of our clients. Our dedicated teams are adept in specific technologies, ensuring seamless integration of skills and delivering reliable, scalable, and high-quality software solutions aligned with our clients' preferences. Bespoke Dedicated Teams: Crafted to meet your specific needs and technology preferences, our dedicated teams are committed to delivering top-notch software solutions. Offshore Software Development: Accelerate your software development and scale up quickly with our 12+ years of expertise in offshore development. Quality Assurance Services: Ensure the quality of your software products with our dedicated teams of experienced QA professionals. IT Staff Augmentation: Overcome skill gaps with our client-centric software team, offering staff augmentation services. Expert Software Services: Unlock our capabilities in custom software development, product development, and quality assurance. Mission and Vision: Our mission at TECUNIQUE is to be the catalyst for our clients' success in the dynamic domain of software development. Rooted in our core values of respect, authenticity, and responsibility, we strive to ease the software outsourcing experience, reducing both time and cost to market for our clients. We envision ourselves as the leading Indian software services company, renowned for our unwavering commitment to excellence and innovation. www.tecunique.com

TECUNIQUE: Success Stories: IT Service provider

mohitmore19

%in Lydenburg+277-882-255-28 abortion pills for sale in Lydenburg

masabamasaba

Conference: Engage2024 in Antwerp Type: Workshop Speakers: Florian Vogler, Henning Kunz, Christoph Adler Title: Navigating the Future with The Hitchhiker's Guide to Notes and Domino 14 Abstract: Embark on an exhilarating journey with industry trailblazers Florian Vogler, Henning Kunz, and Christoph Adler in this not-to-be-missed workshop at the forefront of the tech universe. Get ready for a thrilling kick-off as we navigate the current state of the HCL universe, setting the stage for an exploration of the groundbreaking Notes and Domino 14. Discover the latest enhancements and revolutionary features that will redefine your experience. In this interactive session, unlock a treasure trove of tips and tricks to elevate your utilization of version 14, both with and without the game-changing panagenda MarvelClient. Brace yourself for also diving into Nomad, Nomad Web, and VoltMX, expanding your horizons in the expansive HCL landscape. Be a part of this exclusive opportunity to stay ahead in the ever-evolving world of HCL technologies. Your journey to mastering Notes and Domino 14 begins here. And remember, in the spirit of intergalactic exploration, don't forget to bring your towel!

W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...

panagenda

We specialize in Psychic Readings, Psychic Love Spells, Binding Love Spells, Obsession Spells, Voodoo Spells, Lottery Spells, Marriage Spells, Black Magic Spells, Palm Readings & much more. Are you depressed? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? We perform this come-to-me love spell that works instantly with the aim of bringing back the victim to the person performing the magic. Have you lost your lover? Do u need to solve any relationship problem? Contact the powerful spells caster chief kule with love spells that work overnight and love spells that really work. Have you found yourself infatuated with a special someone you think could be the one? Are you looking for a spell to provide them with a nudge in the right direction? Or maybe the spell you cast didn’t achieve the results you were hoping for? Whether you’re new or versed in the ways of spell casting, we’re here to help. Today we’re going to provide you with a detailed guide on the types of love spells to cast. Not only that but there’s something for those who wish to find outside advice from more advanced spell casters. We’re also going to provide you with the top sites available to help you with your dilemma. Let’s begin our journey by educating ourselves on love magic and what a real love caster looks like. Love Magic and Love Casters Love magic made its first appearance back in Ancient Egypt and has been an active practice since. This type of magic is a branch of traditional magic and can be practiced in various ways. Typically the more common use of love magic is through the work of spells, but other methods look like Charms Rituals-LOVE Potions-Dolls and even Amulets If you are interested in becoming a love caster, be prepared for what’s to come. A genuine love caster knows that the art of love casting is no easy feat and shouldn’t be done casually. You should know that not only does it require you to be gifted spiritually, but you must be ready to serve others. Someone who is considered a real love caster has experience in all manner of spells, no matter the difficulty. Training yourself in attraction, commitment, and marriage spells is an excellent place to start. But this by no means will make you a professional. Practice your craft and expand your knowledge; understand that you will possess the ability to help others in time truly. Types of Love Spells What better way to start broadening your experiences with love spells than by learning more about them? These spells work like just about any other spell. Simply apply your intention, use a medium (sigils, mantras, candles, or charm bags), and top it off with establishing the belief that you will receive what you want. So what kind of spells are available and which ones suit your needs the best? Let’s take a look at the many options you have at your disposal.

%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...

masabamasaba

Software Quality Assurance Interview Questions

Arshad QA

Define the academic and professional writing..pdf

PearlKirahMaeRagusta1

8257 interfacing 2 in microprocessor for btech students

HimanshiGarg82

(Vivek)Call Us, 8448380779,Call girls in Delhi NCr – We Offer best in class call girls. escort Service At Affordable Price At low Rate with Space Night 8000 We Are One Of The Oldest Escort and Call girls Agencies in Delhi. You Will Find That Our Female Escorts Are Full Of Fun, Sexy And They Would Love Enjoy Your Company. We Have A Fantastic Selection Of Escort Ladies Available For In-Calls As Well As Out-Calls. Our Escorts Are Not Only Beautiful But All Have Great Personalities Making Them The Perfect Companion For Any Occasion. In-Call:- You Can Come At Our Place in Delhi Our place Which Is Very Clean Hygienic 100% safe Accommodation. Out-Call:- You have To Come Pick The Girl From My Place We Are Also Provide Door Step Services (Delhi Ncr, Noida, Gurgaon, Faridabad, Ghaziabad Note:- Pic Collectors Time Passers Bargainers Stay Away As We Respect The Value For Your Money Time And Expect The Same From You Hygienic:- Full Ac room And Clean Rooms Available In Hotel 24 * 7 Hourly In Delhi NCR More Details, With WhatsApp Number, +91-8448380779

call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️

Delhi Call girls

VTU technical seminar 8Th Sem on Scikit-learn

AmarnathKambale

%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein

masabamasaba

%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...

masabamasaba

%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...

masabamasaba

+971565801893 Mtp-Kit (500MG) Prices » Dubai [(+971565801893**)] Abortion Pills For Sale In Dubai, UAE, Mifepristone and Misoprostol Tablets Available In Dubai, UAE CONTACT DR.Leen Whatsapp +971565801893 We Have Abortion Pills / Cytotec Tablets /Mifegest Kit Available in Dubai, Sharjah, Abudhabi, Ajman, Alain, Fujairah, Ras Al Khaimah, Umm Al Quwain, UAE, Buy cytotec in Dubai +971565801893''''Abortion Pills near me DUBAI | ABU DHABI|UAE. Price of Misoprostol, Cytotec” +971565801893' Dr.DEEM ''BUY ABORTION PILLS MIFEGEST KIT, MISOPROTONE, CYTOTEC PILLS IN DUBAI, ABU DHABI,UAE'' Contact me now via What's App…… abortion Pills Cytotec also available Oman Qatar Doha Saudi Arabia Bahrain Above all, Cytotec Abortion Pills are Available In Dubai / UAE, you will be very happy to do abortion in Dubai we are providing cytotec 200mg abortion pill in Dubai, UAE. Medication abortion offers an alternative to Surgical Abortion for women in the early weeks of pregnancy. We only offer abortion pills from 1 week-6 Months. We then advise you to use surgery if its beyond 6 months. Our Abu Dhabi, Ajman, Al Ain, Dubai, Fujairah, Ras Al Khaimah (RAK), Sharjah, Umm Al Quwain (UAQ) United Arab Emirates Abortion Clinic provides the safest and most advanced techniques for providing non-surgical, medical and surgical abortion methods for early through late second trimester, including the Abortion By Pill Procedure (RU 486, Mifeprex, Mifepristone, early options French Abortion Pill), Tamoxifen, Methotrexate and Cytotec (Misoprostol). The Abu Dhabi, United Arab Emirates Abortion Clinic performs Same Day Abortion Procedure using medications that are taken on the first day of the office visit and will cause the abortion to occur generally within 4 to 6 hours (as early as 30 minutes) for patients who are 3 to 12 weeks pregnant. When Mifepristone and Misoprostol are used, 50% of patients complete in 4 to 6 hours; 75% to 80% in 12 hours; and 90% in 24 hours. We use a regimen that allows for completion without the need for surgery 99% of the time. All advanced second trimester and late term pregnancies at our Tampa clinic (17 to 24 weeks or greater) can be completed within 24 hours or less 99% of the time without the need surgery. The procedure is completed with minimal to no complications. Our Women's Health Center located in Abu Dhabi, United Arab Emirates, uses the latest medications for medical abortions (RU-486, Mifeprex, Mifegyne, Mifepristone, early options French abortion pill), Methotrexate and Cytotec (Misoprostol). The safety standards of our Abu Dhabi, United Arab Emirates Abortion Doctors remain unparalleled. They consistently maintain the lowest complication rates throughout the nation. Our Physicians and staff are always available to answer questions and care for women in one of the most difficult times in their lives. The decision to have an abortion at the Abortion Clinic in Abu Dhabi, United Arab Emirates.+971565801893

+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...

Health

The title is not connected to what is inside

shinachiaurasa2

Abortion Clinic And Abortion Pills For Sale in Oman MEDICAL ABORTION PILLS FOR SALE in Fujairah , Ajman, Al Ain Quality Abortion services provided by specialists. Women's health problems. A same day service, safe, legal & pain free Abortions. From 1 week to 5 months. We use tested and approved pills. It's 100% guaranteed & safe. No side effects at all. We also do free womb cleaning and free pills delivery. We sell pregnancy termination pills {ABORTION PILLS} Our service is ever private with all our clients. Call/WhatsApp:

%+27788225528 love spells in Vancouver Psychic Readings, Attraction spells,Br...

masabamasaba

%in ivory park+277-882-255-28 abortion pills for sale in ivory park

masabamasaba

Recently uploaded (20)

AI & Machine Learning Presentation Template

Shapes for Sharing between Graph Data Spaces - and Epistemic Querying of RDF-...

Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...

SHRMPro HRMS Software Solutions Presentation

TECUNIQUE: Success Stories: IT Service provider

%in Lydenburg+277-882-255-28 abortion pills for sale in Lydenburg

W01_panagenda_Navigating-the-Future-with-The-Hitchhikers-Guide-to-Notes-and-D...

%+27788225528 love spells in new york Psychic Readings, Attraction spells,Bri...

Software Quality Assurance Interview Questions

Define the academic and professional writing..pdf

8257 interfacing 2 in microprocessor for btech students

call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️

VTU technical seminar 8Th Sem on Scikit-learn

%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein

%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...

%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...

+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...

The title is not connected to what is inside

%+27788225528 love spells in Vancouver Psychic Readings, Attraction spells,Br...

%in ivory park+277-882-255-28 abortion pills for sale in ivory park

Semantic Equivalence of e-Commerce Queries

1. Semantic Equivalence of e-Commerce Queries Aritra Mandal and Daniel Tunkelang and Zhe Wu eBay Inc.

2. Search query != search intent. ● Information retrieval researchers worry about queries that map to multiple intents. jaguar or ? ● Practitioners worry more about multiple queries that map to the same intent. lightning to 3.5mm iphone to aux

3. Equivalent queries should yield equivalent experiences. Recall? CTR? Conversion Rate? ... ? or

4. Opportunity to increase recall while preserving precision. Similar but not equivalent intent.

5. High-level strategy to leverage query equivalence. Map queries to vectors. Store in nearest-neighbor database. (i.e., optimize for user or business outcome)

6. Two strategies for recognizing equivalent queries. ● Surface Similarity ○ Variation in inflection, word order, compounding, noise words. black tshirts for men = mens black t-shirt = ● Behavioral Similarity ○ Queries lead to engagement with equivalent or similar results. lightning to 3.5mm = iphone to aux =

7. Introducing the “bag of documents” model.

8. Query vectors are centroids of associated product vectors ► ► [0.13, 0.81, … ] [0.09, 0.75, … ] … ► [0.11, 0.79, … ] [0.13, 0.81, … ] [0.09, 0.77, … ] … ► [0.12, 0.78, … ] ► cos > 0.98 black tshirts for men mens black t-shirt

9. Works well, but only for head and torso queries. ● Offline approach works for queries with enough engagement history. ● Would be expensive to compute aggregates of result vectors online. ● Still, head and torso queries tend to represent a large fraction of traffic.

10. Train online sentence transformer model for tail queries. ● Train using (query1, query2, similarity) triples from offline model. ● Oversample similar query pairs to increase sensitivity where it matters. ● Fine-tune a pre-trained micro-BERT sentence transformer model. ● Concatenate the output of a query classifier to the query keywords.

11. Architecture for Online Query Similarity Model

12. Results Model Dataset Name Pearson’s correlation query-sim-ecom eBay Internal 0.87 query-sim-ecom ESCI query-query 0.85 all-MiniLM-L12-v2 ESCI query-query 0.68 Query 1 Query 2 cosine hdmi to galaxy s8 s9 hdmi 0.9993 movie money prop money 0.9995 cassette adapter for iphone tape to aux 0.9993 Examples from ESCI of queries with low surface but high behavioral similarity:

13. Summary ● Queries with equivalent intent should yield equivalent experiences. ● Query similarity can increase recall while preserving precision. ● Signals can come from either surface or behavioral similarity. ● Offline bag-of-documents model: queries as means of product vectors. ● Fine-tune online Micro-BERT sentence transformer model for tail queries. ● It just works!

Semantic Equivalence of e-Commerce Queries

Recommended

Recommended

More Related Content

Similar to Semantic Equivalence of e-Commerce Queries

Similar to Semantic Equivalence of e-Commerce Queries (20)

More from Daniel Tunkelang

More from Daniel Tunkelang (20)

Recently uploaded

Recently uploaded (20)

Semantic Equivalence of e-Commerce Queries