Developed an AI-powered Essay Grading Solution for an EdTech company, resulting in an 85% reduction in average grading time

The customer’s platform is used by Graders and Trainers for teaching course management, student learning and evaluating progress. The trainers and graders regularly encounter many issues around grading assignments. Grading essays is a laborious and time-consuming task for educators, and it becomes tough for the grader to view all assignments with the same level of attention as the first few. It also becomes tedious to keep track of different sets of Rubrics for assessing the different types of assignments.

Critical challenges:

• Evaluating is time-consuming for educators - Taking up an average of 45 mins
• With the increase in the number of grading assignments, scaling up was a challenge. More personnel were needed, which could not be a sustainable option if there is a fluctuation in demand.
• Maintaining consistency and objectivity while evaluating essays is difficult
• Lack of accuracy and reliability in manual grading

​​Instead of spending a significant amount of time reviewing each essay and providing detailed feedback, they wanted to automate certain tasks, freeing up more time for educators to focus on higher-level tasks, including teaching and student learning. The goal was to build an Essay Grader MVP by leveraging GenerativeAI tools like ChatGPT. The proposed solution could evaluate the assignments faster by providing a suggested grade according to the rubrics provided.

The aim was to empower graders and trainers to improve efficiency and productivity by developing AI-powered tools. The assistive technology can reduce human error, giving consistent, repeatable results with remarkable performance.

They partnered with us, considering our vast expertise in building products and expert systems driven by AI and ML technologies like deep learning models (CNNs, RNNs, LSTMs and SOMs), Natural Language Processing (NLP) using open-source and proprietary LLMs (GPT-4, ChatGPT), TensorFlow, PyTorch, computer vision using DALL-E 2, OpenCV, CUDA, Keras, Midjourney and Stable Diffusion and speech recognition.

We wanted to integrate AI tools with advanced natural language processing capabilities. This could help analyze and understand complex topics. The goal of the platform was to build a solution that could evaluate the essays on the parameters, including structure, coherence and rubrics to meet the requirements of different academic levels and disciplines.

Keeping in mind the constricted timeline, we quickly set up a team comprising Backend Engineer, Frontend Engineer, Data Scientist, and Team Leads for the project. They kickstarted the project by creating a comprehensive roadmap and evaluating the potential challenges that needed to be tackled.

They determined the right tech stack to be used for both The Grader Dashboard and Trainer Dashboard. ‍

(I) Roadmap to visualise the different dashboards for Essay Grader MVP

a) Grader Dashboard

• The graders can upload an essay and provide additional information, including the type of essay (Argumentative, Research papers etc.) and select a predefined rubric from the prepopulated list.
• They can also provide their own weightage to each parameter in the rubric.
• This information can then be used to grade the essay and provide comments.
• The proposed Essay Grader MVP should process the essay based on the information provided and produces a relevant grade which is available on the Grader’s Dashboard, along with the option to be able to edit them.
• The grader can then change the grade or add comments depending on their perspective and experience.
• Also, a feedback section was needed where the graders could provide their feedback on the accuracy of the grading system.

b) Trainer Dashboard:

• The trainer dashboard was to be built for the trainers (experienced SMEs).
• They would have access to the grades provided by the graders in case of a discrepancy reported in the grade (evaluated by the system) by the graders.
• The trainers could then re-evaluate it and check whether there are any actual errors or not and accordingly provide feedback.
• This feedback could then be used to train our Grader system to provide better and more focused evaluations in the future.

(II) Our approach to building a cutting-edge AI-powered solution

a) Identifying the right tools:

We identified the best AI model and different tools required by carefully considering the budget and the requirements. Training the AI model can be quite cost-intensive, and choosing the right AI training data is crucial for any successful application. The team identified the size of the training data, accurate information and data depicting all the test cases which will be used to train the Grader model driven by AI.

• We conducted a tech review and proposed a product workflow, architecture design as well as UX design.
• We designed and implemented an updated DB schema (models: essays, tenants, grading, feedback, comments, etc.) which was then reviewed by product owners and stakeholders.
• We explored and prepared training data, including essay types, rubrics configuration, and organisation configuration.

b) Integration and Training:

We set up a proper testing environment for the AI model, which was totally detached from the main working model. This ensured seamless training and testing of the model from time to time to make it more efficient.

• We conducted initial training and fine-tuned the base model with the training data (Curie/DaVinci).
• We set up and implemented API integration with ChatGPT.
• We defined the testing strategy - unit, regression and integration.
• Essay input UI and API was implemented for uploading files, additional details and rubrics selection.
• We then integrated essay grading and commentary request/response UI and API.
• The feedback section for the overall model grade response (upvote/downvote/comment) was added.
• Infrastructure setup followed by the deployment of the model in the Staging/UAT environment was implemented.
• Post-deployment demos and review with stakeholders/product owners were conducted.

c) Monitoring and Setting the feedback loop:

Once we set up the Grading tool, regular monitoring was needed to evaluate the system. We architectured a good trainer feedback system so the trainers could check the discrepancies reported by the graders. It further checked whether the feedback was precise and accurate, which could be used to make the Trainer model better.

• We set up a system to store grades and reports (supported file format: DOCX file), which could be downloaded when required.
• Admin essay management UI/CRUD operations interface was integrated. Through this, Trainers could create new rubrics and edit existing rubrics.
• The feedback loop was set up to ensure all the discrepancies are fed back to the system in order to improve evaluation in the future.

We then defined and implemented the KPIs for internal analytics and business reporting. To ensure that all stakeholders are equipped to use the new tool, we developed training and documentation for graders/trainers. We conducted regular demo/review sessions with stakeholders/product owners and thorough end-to-end testing was done before go-live.