Going Beyond Vector Search to Help Match Jobs to People
August 13, 2024
Hint: Artificial Intelligence has to be biased to be effective.
In today’s job market, finding the right candidate for a role is essential for organizations aiming to thrive. Traditional keyword-based search methods often miss the subtle connections between job descriptions and applicant profiles. While vector search has improved candidate matching by considering semantic similarities, we can go even further. In traditional e-Commerce search, the ranking (i.e. Relevancy) is tuned based on past purchases. Given analytic data like add-to-cart and checkouts, we can create a feedback loop and implement learning to rank to favor what someone is looking to buy when they search for a particular query term.
In this way, we are adding bias to the search experience. This is often an organization’s “special sauce” or secret formula. By embracing the idea that artificial intelligence must be biased to be effective, we can refine the process of matching jobs to people. After all, matching candidates is about more than just aligning tokens — it’s about understanding the unique blend of skills, experiences, and potential each individual brings.
Not all tokens are equal and embrace ‘bias’ for Effective AI
The notion that AI must be biased might seem counterintuitive, but when you examine larger amounts of data, you must create some weighting algorithm. In past projects, this has been expressed in the form of Recency, Frequency, and some measure of strength.
Effective AI should be designed to prioritize certain characteristics and experiences that align with the goals and culture of an organization. This purposeful bias helps filter out irrelevant candidates and focuses on those more likely to succeed and thrive in specific roles.
Understanding the Limitations of Vector Search
Vector search transforms text data into mathematical representations, allowing systems to identify semantic similarities. This technique has enhanced applicant searching by enabling recruiters to find candidates whose resumes align more closely with job descriptions, even if they don’t use the same keywords. However, vector search has limitations:
- Contextual Misunderstandings: Vector search can misinterpret context, particularly in specialized fields where similar terms may have different meanings.
- Lack of “Page Rank”: The general vectors are designed to match and predict sentences. They are not designed to weight a particular sentence or passage.
- Scalability Issues: Handling large volumes of data with vector search can be computationally expensive and may require significant infrastructure investments.
Better Strategies for Matching Jobs To People
The focus should be a ranking in addition to retrieval. Vectors can help with retrieval by increasing recall.
So this may return a candidate or a job, but if the recruiter or candidate isn’t aware of why these matches occur, it may cause them to be passed up.
I remember having to plead with a recruiter to submit my resume because T-Sql (the SQL language of Microsoft SqlServer) was not on my resume but SqlServer was.
1. Custom Scoring Models
Implementing a system of multiple ranking phases can help sift through he results. In Vespa, for example, there is a means to phased ranking where the results are recomputed before returning. This allows you to reduce the ‘haystack’ before adding custom scores and weights.
2. Knowledge Graphs
Knowledge graphs represent information in a network of entities and relationships, providing a more structured way to capture complex data. By integrating knowledge graphs into applicant searching, recruiters can visualize connections between candidates’ skills, experiences, and job requirements. This approach can highlight indirect relationships that vector search might miss, such as complementary skills or industry-specific expertise.
3. Using AI to Normalize Candidates
AI can be used to reformat a candidate’s CV into a more structured representation. Once in a more structured form, it can be converted into a vector graph more easily.
4. Machine Learning and AI Models
Incorporating machine learning and AI models can further enhance applicant searching by predicting candidate success and cultural fit. These models can analyze historical hiring data to identify patterns and trends, enabling recruiters to focus on candidates who are more likely to thrive in a given role. AI-driven recommendations can also suggest potential candidates who may have been overlooked by traditional search methods.
Conclusion
While vector search has brought significant improvements to Applicant and Job searching, there is still potential for further enhancement by integrating advanced technologies and methodologies. By leveraging hybrid search models, knowledge graphs, NLP enhancements, machine learning, and personalized experiences, recruiters can refine their search processes and uncover candidates who are not only qualified but also well-suited to their organizational culture and goals. As the job market continues to evolve, staying ahead of the curve with innovative search strategies will be key to attracting and retaining top talent.