AI in Design

Assignment 3 Part 1

Updated Nov 13, 2024

It has come to my attention some of the students were following the links from blackboard and not on this site. To rectify this mistake, I have pushed the deadline by a week. Additionally, I have added a short descriptions links and short video on how to use and evaluate the interfaces below.

Back to Assignment 3 Part 1

You are tasked with creating an application that uses Large Language Models to satisfy certain interactive objectives you specify.

The Objectives are outlined by you and are evaluated by your peers and course staff.

Your tasks include:

  1. Select a Large Language model for solving your objectives.
  2. Create ways to benchmark the performance of your model for your “objectives”.
  3. Design the interactive scenario for your application and explain the setup principles behind this interaction scenarios/behaviors.
  4. You would break down the interaction scenarios into multiple tasks. That is, LLMs are not able to do everything at once. Prompt engineering isn’t enough. There are supporting structures that you would need to design/anticipate. Each of this subtasks: could be a function to be done in a different AI application, or LLM or even human should be triggered by the LLM or through the interaction with the LLM.

Assignment Lessons:

  1. Converse with Multiple LLMs and develop a sense of how they perform on different tasks.
  2. Craft a testing document that you could use to benchmark the performance of your LLM across different objectives.
  3. Design the interactive scenario for your application and explain the setup principles behind this interaction scenarios/behaviors.
  4. Find the parameters that you need to set for your LLM to perform well on your objectives.

Assignment Support

As we had done in the class; we have four LLMs setup in different ways to help you get started organised in steps to allow you to focus on each of the objects.

  1. Talk to the multiple LLMs at time to get a sense of how they perform on similar prompts system prompts. Claude, OpenAI, Qwen and Llama Link
    • You could ofcourse evaluate other LLMs as well. But due to logistical reasons, we are only able to support these four or few more under fireworks.ai and Claude/Gpt4o for obvious popularity reasons.
  2. Talk to one LLM at a time with fine control over the parameters with the system prompt, setting the temperature, max_tokens, top_p, frequency_penalty, presence_penalty to indentify the best parameters for your question Link

  3. (REQUIRES SUBMISSION - Portal Coming Soon)Prepare a testing “JSON/TXT” file that you could use to submit to each LLM and get responses for and save the response. Link An explanation video is given above

  4. Preliminary Design of your application and interaction scenarios bot design and Submission - Nov 23

Submit upto this on blackboard

Blackboard link: Link

Additional Prompt Engineering Resource

(NOT GRADED)Part 2 of the Assignment (To be released on Nov 15)

Tying Everything Together / Building It

We will upload demonstrative videos for the sections below.

This section is not graded. But it is a good way to tie everything for those who are interested in seeing how everything comes together.

The things we have covered till now present us with

In this part we will try to see how we can call use LLMs to call those desparate AI so that they could do something on behalf of vairous inputs for using an illustrative example from your submissions and our own examples.

  1. Function calling and different Vision LLMs example illustration and integration.

We show an exammple of LLM calling external tools based on our conversation to generate an audio, image. The overall example here is “audio”/”video” could be any functional representation.

  1. Integration with Audio/Video/Avatar/Application and Moving Beyond Text

All up till now, we built a chatbot and the primary model of interaction was text. But using the similar AI frameworks we discussed on Image/Video tutorails we could replace the modality of interaction.

In this section, we will use Whisper and CoqTTS to use audio as a modality of interaction.

  1. Avatars and Other Applications

In this part, we will go to use creative engines like Unreal Engine / HeyGen and similar tools like we have explored in the class to drive avatars and see the experience of using AI in various applications.

You could of course use other LLM solutions as well. If you are interested in implementing your submission for part 1 using this options, you can reach out to us and we can suggest some options of bringing it to reality after your examinations.

Would further continue on 5/6/7 to integrate Audio/Video/Avatar/Application integrations using the tutorials given previously we will see your initial submissions to provide appropriate platforms for this expressions.

Examples of Applications

For instance:

Project 1: Objective: Friendly Robot/Avatar that can have a conversation with you.

Project 2: Objective: Plant Cultivation and Stress Reliefer

We could choose to design a chatbot that guides plant cultivation to relieve stress among young people, you could use plant cultivation guidance as a medium to address emotional stress. This choice could be based on the healing properties of plants and the bonding effect of sharedcultivation, facilitating user engagement with the chatbot.

Project 3: Objective: Customer Support Robot

We could choose to design a chatbot that answers questions about a company/product with focus given some information.