​

Introduction

Collinear Flex Judges help evaluate responses based on specific criteria or use cases. They are tailored to unique evaluation requirements, industry standards, or user-defined metrics. Collinear Flex Judges are versatile and can be designed to assess responses in various contexts, such as competitions, assessments, or automated content moderation.

They ensure that response are:

• aligned with specific evaluation criteria
• adhere to specific standard
• meet user-defined metrics

You would have to initially provide some human annotations to train the custom judge. The custom judge will then be created based on the annotations provided.

​

Steps to Create a Custom Judge

​

Select Annotations In Curated Data

Go to the Curated Data section and select the annotations you want to use to train the custom judge.

​

Click on Create Judge

Click on the “Create Judge” button to start the process of creating a new judge.

​

Select Annotations To Use

Clicking on the “Create Judge” button will open a modal where you can select the type of annotations.

• Judge Type: Select the type of judgement you want to use:
  • Collinear Flex - Use Collinear Flex Judge
  • Prompted Model - Use Other Prompted Models
• Judgement Type: Select the type of judgement you want to use:
  • Human Feedback - Use human annotations
  • Judge Feedback - Use AI annotations
• Scoring Criteria: Select the type of scoring criteria you want to use: This auto populates based on the judgement type selected and scoring criteria available in all the selected rows. It could be in the form of binary_user, likert_user,binary_avg, etc. CLick on “Submit” to continue.

​

Configure Custom Judge

The next step is to configure the custom judge. You will provide the following details:

• Prompt - We provide two options:

  • Vanilla - You can choose to use off the shelf Collinear Flex Judge. This judge is more suited on safety.
  • Auto Generate Scoring Criteria - The instructions will be generated based on the examples provided. The scoring criteria is auto generated based on the examples provided. And Click on the “Generate Prompt” button.

So now we have the configuration in place! Click on “Continue” to proceed.

​

Add Judge Name

Name your judge according to your preference and click on “Create Judge”.

​

Judge Created

Your Collinear Flex judge has been successfully created. You can now view, edit, test, or delete the judge from the Judges Dashboard.