Korean AI Model Rater

Remote

Contracted

Developmental

Experienced

AI Model Rater, Korean (Coding knowledge required, with focus on APIs)

Join our team of Korean speakers with coding knowledge! Apply now to help improve the next generation of AI models' tool-use capabilities.

Location: Remote, global (not US-based)
Status: Project-based
Pay: USD $24 per hour
Benefits: Flexible schedule

To ensure eligibility for this project, applicants must be international freelancers. Unfortunately, we are unable to hire U.S. citizens, Green Card holders, or candidates currently residing in the U.S. and also in China.

Project Requirements:

Advanced/Expert fluency in Korean language, as spoken in the South Korea region (ISO: [ko_kr])
Strong comprehension of English (read and write)
Strong conceptual knowledge of APIs and digital services (ability to understand how AI tools interact with external apps like search, maps, or calendar, without needing to write code).
Ability to follow complex technical guidelines precisely.

The Project:
We are seeking detail-oriented and motivated Korean speakers to join our AI LLM evaluation project. This project focuses on assessing the accuracy and quality of AI-generated responses that rely on integrated applications and tools (APIs).

The work involves evaluating a variety of AI-generated outputs, specifically paying attention to the correct and appropriate use of tool outputs (e.g., verifying that a proposed calendar event is correct, or a search result is relevant). This task evolves reading and judging the quality of API code. You must be able to adapt to various API standards by efficiently reading the provided API documentation.

The sample conversations are displayed in Korean (as spoken in South Korea). The entire evaluation must be written in English.

All work must be completed in accordance with the detailed guidelines provided upon acceptance. Please note that this work is project-based, and the workload may vary. There is no guarantee of a minimum or maximum number of hours.

Duties:

Analyze and rate AI-generated responses and their associated tool outputs against a detailed set of quality guidelines.
Verify that the AI successfully integrates information from different digital services (APIs) as required by the prompt.
Evaluate the accuracy, completeness, and overall quality of the AI's response and tool execution.
Follow provided instructions to achieve task goals, with all evaluations recorded in English.

Engagement Requirements:

All participants must have, or be willing to create, an Upwork account. The project will be managed via the Upwork platform.
While participating in this project, adherence to the confidentiality terms outlined in the Upwork User Agreement is required. Any information accessed or received related to this project is confidential and may not be shared or disclosed to third parties.
Applicants must pass a quick skill verification assessment (MC, duration ~10 minutes).
Work must be performed on participant’s own devices.
You must use your own laptop, and smartphone (if required).
A Gmail account may be required to access certain tools. If you do not have a Gmail account, you must be willing to create one.
Using AI during the assessment and work is STRICTLY forbidden, and results/inputs will be monitored to detect AI usage. Any detections will result in immediate ban from any future engagements.

About Us:

As a global data company, Productive Playhouse “PPH”, is pioneering our approach to language and data services, while incorporating their roots as a production company. Originally creating content to support children's language acquisition, our commitment to excellence, forward-thinking strategies, and world-wide cultural experience has proven key for delivering exceptional service.

Originally founded as an educational production company, Productive Playhouse made a mark with our award winning children’s series, which taught fundamental subjects through engaging and effective programming. This early success paved the way for our evolution into a comprehensive data services provider.

Since 2011, Productive Playhouse has expanded rapidly to offer an extensive suite of data services. Our current offerings include transcription, translation, linguistic analysis, rating, systems testing, localization, field and studio recording, language skill verification, and specialized data handling with a focus on sensitivity and diversity. Our commitment to innovation means we continually enhance our service portfolio to meet the evolving needs of our clients.

At Productive Playhouse, we are proud of our reputation for addressing complex challenges with agility and delivering premium secure data solutions across diverse environments. Our dynamic team is dedicated to maintaining the highest standards and ensuring exceptional service every time.

Disclaimer

The job description provided is designed to convey information essential to understanding the scope of the position and the general nature of the work performed. It is not intended to be an exhaustive list of duties, responsibilities, or qualifications associated with the job. Productive Playhouse reserves the right to modify or revise the job description as necessary

Productive Playhouse is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive work environment for all employees. Employment with Productive Playhouse is at-will, meaning that either the employee or the employer can terminate the employment relationship at any time, with or without cause or notice.

All offers of employment at Productive Playhouse are contingent upon the candidate’s ability to provide valid documentation of identity and eligibility to work in the United States or relevant hiring location. Productive Playhouse participates in E-Verify.

Productive Playhouse provides reasonable accommodation for qualified individuals with disabilities. If you need assistance or accommodation due to a disability, please contact the Human Resources Department.

Apply for this position

Required*

First Name*

Last Name*

Email Address*

Phone*

Address

Resume*

We've received your resume. Click here to update it.

Attach resume or Paste resume

Attach resume as .pdf, .doc, .docx, .odt, .txt, or .rtf (limit 5MB) or Paste resume

Paste your resume here or Attach resume file

Which language are you applying for?*

Do you meet C2 proficiency in this language, and what is your English proficiency level?

On a scale of 0 to 10, how would you rate your knowledge of APIs?*

This project requires participants to have their own smartphone and laptop. Please select your device specifications below.*

This project may require participants to record short video clips that could include faces or objects. These requirements vary by project.

Are you comfortable participating in projects that may involve video recordings?*

How many AI projects have you worked on previously?*

How many years of verifiable experience do you have in transcription and/or translation?*

What country and city do you live in?*

This project is only open to international freelancers who are not U.S. citizens or Green Card holders. Are you a U.S. citizen or Green Card holder?*

If hired, you will be asked to complete an identity check that involves sending a photo of yourself and an image of your identity document. Is this acceptable to you?*

Do you have an Upwork account? If yes, please provide the link.

Are you willing to make an Upwork account, if you do not already have one?*

Have you ever worked for Productive Playhouse before?*

Please check your spam or junk folder for communication regarding your application. Updates may be sent from email addresses with the following domains; productiveplayhouse.com, applytojob.com, jazzhr.com, jazz.co* OK

Submit Application