I have cleared the Telus International (Formerly Lionbridge) Internet Assessor Exam Part 3 (the same role is named Rater for USA applicants, Appen called it as a Project Yukon, RWS called it as a Search Engine Evaluator and Welocalize called it as a Search Quality rater. Exam is same for all these roles as client is Google who conducts exam for this role.) early last year. It’s probably the hardest part of the exam. First of all, you need to analyze queries to understand user intents and needs fully because ratings should be given according to the user’s intents. I think people fail this part because they are approaching the tasks ignoring the user intents and needs. Secondly in this part of the exam, each task consists of 8-10 result blocks to rate their both Page Quality and Needs Met, within 1 hour. Remember in Part 2 we are giving PQ rating for single web page and there is no query. Here we need to understand query first and need to give PQ rating which is independent of query or its intent and Needs Met rating which depend on understanding query and user intent. So in all you are evaluating 8 to 10 web pages for PQ and Needs Met rating. Many people fail in the Part 3 because they are not ready for such a huge task to complete as compare to Part 2. Key to pass Part 3 is understanding Needs Met properly and be ready to evaluate 8 to 10 result blocks in a hour time mentally.
Part 3 (Needs Met Part)
Now this is most difficult part of the exam. Most of people fail to clear this part. Even if you fail don’t get disheartened normally you will be offered second chance for the exam which is usually 1 week later. But you need to start again from Part 1 i.e. you need to complete again all the three parts sequentially which is good because you need to apply the knowledge of whole guidelines to clear the Part 3. Here again as mentioned above you need to clear 7 tasks and duration is 1 hour each. You can take break after submitting each task and is highly recommended. Keep 2 full days for Part 3.
Part 3 is focussed on Needs Met rating. As mentioned above guidelines are having separate Needs Met section. Rest of the day 4 I have gone through Needs Met section again clicked on all the links for the examples given. I have planned to complete 3 tasks on Day 5 and 4 tasks on Day 6, keeping Day 7 reserve.
Part 3 is totally different basically it is actual work format, now you will be evaluating the whole task. Understanding the query is the first step in evaluating the task. Remember, a query is what a user types or speaks into a mobile phone. Many queries have more than one meaning. For example, the query [apple] might refer to the computer brand or the fruit. We will call these possible meanings query interpretations. The dominant interpretation of a query is what most users mean when they type the query. Not all queries have a dominant interpretation. The dominant interpretation should be clear to you, especially after doing a little web research. A common interpretation of a query is what many or some users mean when they type a query. A query can have multiple common interpretations. Sometimes you will find less common interpretations. These are interpretations that few users have in mind. We will call these minor interpretations. Remember to think about the query and its current meaning as you are rating. We will assume users are looking for current information about a topic, the most recent product model, the most recent occurrence of a recurring event, etc., unless otherwise specified by the query.
Secondly you need to understand user intent from the query. Many queries have more than one likely user intent. Please use your judgment when trying to decide if one intent is more likely than another intent.
This is what the Needs Met rating slider looks like:
Rating Slider
Fully Meets (FullyM): A special rating category, which only applies to certain queries and results. All or almost all mobile users would be immediately and fully satisfied by the result and would not need to view other results to satisfy their need.
Highly Meets (HM): Very helpful for many or most mobile users. Some users may wish to see additional results.
Moderately Meets (MM): Helpful for many users OR very helpful for some mobile users. Some or many users may wish to see additional results.
Slightly Meets (SM): Helpful for fewer mobile users. There is a connection between the query and the result, but not a strong or satisfying connection. Many or most users would wish to see additional results.
Fails to Meet (FailsM): Completely fails to meet the needs of the mobile users. All or almost all users would wish to see additional results.
The Needs Met rating is based on both the query and the result. You must carefully think about the query and user intent when assigning a Needs Met rating. The Page Quality rating slider does not depend on the query. Do not think about the query when assigning a Page Quality rating to the LP (Landing Page, Web Page). Some results don’t have a Page Quality slider. If a result block has no Page Quality rating slider, you do not have to give a Page Quality rating. If there is a Page Quality slider, please assign a Page Quality rating based on the landing page.
You need to assign p*rn, Foreign Language, Did Not Load, and Upsetting-Offensive flags to result blocks when appropriate. Some rating tasks may ask you to identify Not-for-Everyone results. All flags are query-independent, meaning that they do not depend on the query.
Some queries really only have one meaning. Consider the query [iphone], English (US). There may be different user intents for this query (research iPhones, buy an iPhone, go to the iPhone page on Apple’s website), but all users are basically referring to the same thing: the phone made by Apple, Inc. Some queries truly have different possible meanings. Consider the query [apple], English (US). Some users may want to find more information on the computer brand or the fruit. We refer to these different meanings as query interpretations. When giving Needs Met ratings for results involving different query interpretations, think about how likely the query interpretation is and how helpful the result is.
A very helpful result for a dominant interpretation should be rated Highly Meets, because it is very helpful for many or most users. Some queries with a dominant interpretation have a FullyM result. A very helpful result for a common interpretation may be Highly Meets or Moderately Meets, depending on how likely the interpretation is. A very helpful result for a very minor interpretation may be Slightly Meets or lower because few users may be interested in that interpretation. There are some interpretations that are so unlikely that results should be rated FailsM.
It’s probably the hardest part of the exam. First of all, you need to analyse queries to understand user intents and needs fully because ratings should be given according to the user’s intents. I think people fail this part because they are approaching the tasks ignoring the user intents and needs. A result cannot be automatically useful because of its relevance to the query. You should think like you are the person who wrote that query.
In this part of the exam, each task consists of 8-10 result blocks to rate their both Page Quality and Needs Met, within 1 hour. Remember in Part 2 we are giving PQ rating for single web page and there is no query. Here we need to understand query first and need to give PQ rating which is independent of query or its intent and Needs Met rating which depend on understanding query and user intent. So in all you are evaluating 8 to 10 web pages for PQ and Needs Met rating.
Many people fail in the Part 3 because they are not ready for such a huge task to complete as compare to Part 2. As it is not mentioned anywhere in guideline or in the exam mail. So one need to mentally prepare for this before starting on Part 3. But is it really a huge task? Answer is no. When you will actually start working you will be rating such a task in 10 minute. Here you have 1 full hour. Just keep in mind that there are many things to do in Part 3 as compare to Part 2. Secondly don’t wait for full 1 hour to submit the task. As you are rating 8 to 10 blocks it may happens that 1 or 2 ratings are forgotten to mark. Submit the task on 55 or 56 minute on timer as it may take 1 or 2 minute to submit rating if anything is incomplete the task will not submit and it will highlight that rating in red square. If you fail to submit the task within hour it will get auto submitted. Auto submitted tasks are not considered for evaluation and you will lose the whole marks for that task.
In each task you will have all type of PQ rating as mentioned earlier and all type of Needs Met rating. i.e FullyM, MM, MM, SM, and FaislM. You can also give in between rating. Don’t be generous if in doubt give lower rating. If in doubt between two ratings give in between rating. Most of the results are having tolerance of + or – or both. But here rules are stricter as compare to PQ rating. Fully met is always fully met and no tolerance almost in all cases. Similarly Fails to meet is always fails to meet in almost all cases.
Key to pass Part 3 is understanding Needs Met properly and be ready to evaluate 8 to 10 result blocks in a hour time. I have completed Part 3 on Day 5 and Day6 taking 1 to 2 hour gap in each task. After completing Part 3 you will not get result immediately, if your pass. If you fail you will get mail that you are not successful immediately. But usually they give you second chance and you need to go through all the parts again. If you don’t get any mail, need to wait for 2 to 3 days after exam period is over and then you will notified that you have achieved overall pass they did not share feedback or actual marks.
I received the welcome mail on 30th April. And I started working on 1st of May. It takes 15 days from the application to actual starting working as a Internet Assessor.
While Working
You are allowed to maximum 20 hours per week and minimum 10 hours per week is mandatory when work is available. Sometimes you are allowed to work for 24 to 30 hours per week. I have received this opportunity more than 10 times in a year. There are times when work availability is low and you can’t get full quota of work.
Initially working can be overwhelming you need to grasp many things. But remember you have learned most difficult part of your work while preparing for exam. They will also consider you for first one or two month. You will get used it within a month. In actual work you get many different types of task. Most of them are super easy and fun to work on. Like you will get YouTube video to rate. Audio files to judge for the quality. Every task is having estimated time tagged with it. Remember here you are paid on hourly basis and not on task basis. If you rush through the tasks you will only reducing your chances of getting full quota of work. Vice versa if you have taken more than estimated time for completing task your productivity will be hampered and you will warned for such a practice. Your time sheet can be sent back for corrections. Thumb rule is report time based on estimated time of each task however around 5% variance is acceptable.
Once or twice in a month there will be blind test. Blind test will be based on the task type which are appeared in exam. So guidelines are important while working as well. Guidelines are your rule book. You need to maintain the quality of 85% for first year of your work barring first 3 months of your work. First year onward it is 87%. You will get feedback on blind test every month and you are expected to improve. Continuous bad feedback can kept you on review and you will get very few task and that too are blind test task still you will improve your result to expected level. Sometimes they give you additional learning path and once cleared you will be normal to work on. In month or two by looking at the task you can be able to know whether it is a blind test or not and can proceed with extra care. But it is advisable to consider every task as a blind test and be faithful to yourself while giving judgement. If you know guidelines by heart your judgement will be always in line. If you have passed such a tough exam then working is not going to be difficult. There is no harm in going back to guideline and refreshing your knowledge once in a while.
Exam Assistance – On-job Support
We provide assistance toTelusAssessor/ Rater Exam, Internet Safety EvaluatorWelocalizeSearch Rater Exam,RWS Search Engine Evaluator, Appen ConnectYukon Exam
If you need paid consultation or assistance on Exam/ on-job. Please write to me at techsurbhi@gmail.com orWhatsApp/ Telegramat +91 9960 622 622for guaranteed results and on-job support
Download Guidelines:
Search “General Guidelines – Raterhub guidelines.raterhub.com” in google as the direct link didn’t get hyperlinked in this book for the same.
Application Link For USA Rater :TelusCareersPage
Application Link For Rest of the World:TelusCareersPage
Application Link for Appen Connect:Sign up& Look for Project Yukon
Application Link For Welocalize Rater :*Scout Search Quality Rater
Application Link For RWS :Search Engine Evaluator
How Do I Get Paid?
Pay rate is around $6 per hour for Africa And Asia, $13 per hour for USA, $16 per hour for European countries. Telus International pay by direct payment transfer. Amount will be credited by invoice + 30 days. That is to say your January month payment you will receive between 10th -15th of March. Even if you leave the program you will be receiving all the dues on expected dates.
For Appen Connect payment method is Payoneer.
Download Guidelines:
Search “General Guidelines – Raterhub guidelines.raterhub.com” in google
As direct link didn’t get hyperlinked in this post for the same.
