Difference between revisions of "User:Khatami"

Latest revision as of 10:02, 19 July 2023

2023 Week 1: (May 30th 2023-June 2nd 2023)

Tuesday (5/30)

• Attended REU Orientation
• Got familiar with mentors and cohort
• Met with Dr. Madiraju to learn about my project Assignment
• Contacted IT to activate Marquette ID and Marquette email

Wednesday (5/31)

• Attended Good Research Practice session
• Chatted with Dr. Madiraju about the project assignment once more
• Met Sajjad, the graduate student active on PEER SURE App, and discussed how to build an annotated bibliography as well as important keywords and topics which are most relevant to our current research.
• Met with Christian, another REU cohort member, who will also be working on this project.
• Reviewed the notes and slides from Dr. Brylow's Good Research Practice session
• Some free writing to clear my mind on this project:
  • So far I have prepared a list of main priorities for my literature review and the necessary tools for this project
  • 1.Becoming more familiar with the OUD field and studying the efficacy, format and uses cases for Peer Support in clinical treatment, primary care, and eHealth
  • 2.Reviewing the existing web applications and platforms that offer peer support for substance use disorders
  • 3.Searching for resources to learn how to use C# and generative AI tools and frameworks for web development

Thursday(6/1)

• Initiated a literature review on generative AI for SUD to identify the best practices and the gaps in the field
  • Benefits of peer support groups in the treatment of addiction https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5047716/
  • Lived Experience in New Models of Care for Substance Use Disorder https://www.frontiersin.org/articles/10.3389/fpsyg.2019.01052/full
  • Qualitative Study of Addiction Peer Mentorship in the Hospital https://pubmed.ncbi.nlm.nih.gov/31512181/
  • These papers are not generally focused on web-based Peer Support. However, they were useful for understanding the definition, history, efficacy, and formats of Peer Support. Based on my readings, it appears that Peer Support has been historically based in clinical settings, and only recently (since 2011) there are a few papers mentioning web-based Peer Support. The overall approach to using Peer Support at eHealth and primary care settings is fairly new. I have gathered a few more focused papers on web-based forms of Peer Support which I will begin reading soon.

Friday(6/2)

• Opening a page in Marquette Wiki for Project milestones and goals
• Searching for a tutorial for using Open AI API as well as how How to connect to ChatGPT with C#

2023 Week 2: (June 5th 2023-June 9th 2023)

Monday (6/5)

• Attended RCR Training with Dr. Brylow
• Worked on online RCR modules

Tuesday (6/6)

• Attended REU Group Collaboration
• Continued and finished working on RCR modules
• Began and finished DL2 modules

Wednesday (6/7)

• Attended Technical Writing Session by Dr. Brylow
• Met with Sajjad and Dr. Madiraju to discuss the more specifics of the project including:
  • Clarifying research expectations
  • Discussing the criterion for co-authorship
  • Discussing graduate schools and how to build a successful graduate application
  • Access to C# trainings
  • Access to Open AI prompt engeering course
  • Access to key words and data bases where more relevant papers could be found
• Summarized the results and sharings of the meeting with Sajjad and Dr. Madiraju and sent them to Christian for his review
• Found and began reading more relevant papers for web-based Peer Support:
  • A Pilot Study Comparing Peer Supported Web-based CBT to Self-Managed Web CBT https://pubmed.ncbi.nlm.nih.gov/30489140/
  • Mobile Peer-Support for Opioid Use Disordershttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC7059630/
  • Web-Based Intervention for Returning Veterans with Symptoms of PTSD and Risky Alcohol Usehttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC4219624/
  • Peer Support for Opioid Use Disorders: Feasibility and Acceptability of a Moderated Text-Based Group Chat Application https://www.omicsonline.org/open-access-pdfs/peer-support-for-opioid-use-disorders-feasibility-and-acceptability-of-a-moderated-textbased-group-chat-application.pdf

Thursday (6/8)

• Finished reading the papers with a specific focus on web-based Peer Support from yesterday
• Some general findings from other papers emerged after I finished and reviewed the papers from yesterday and last week. These could be useful for this project:
  • Peer Support mobile or web-based applications use a survey periodically to check for symptoms of opioid (or alcohol) addiction).
  • Traditional forms of treatment for substance abuse face challenges with engaging and retaining the patients through the treatments
  • Peer Support offers a solution by connecting peers with lived experiences of substance abuse with those who are trying to manage and control their addiction.
  • Additionally a sustained recovery from abused substances require constant checkups and following up with treatments which Peer Support groups may help with by providing a community and an opportunity to seek advice, encouragement, and proper treatment without the fear of stigma.
  • Many patients experiencing OUD seek advice and information from online platforms and social media which can be harmful and lead them to misinformation
  • Additionally challenges listed in Peer Support online spaces include:
  • Lack of effective moderation
  • Lack of Common Security
  • Web or mobile-based peer support applications, therefore, can fill these gaps by directly reaching patients and providing them with accurate information while removing the harmful social isolation which brings many to substance abuse in the first place and can draw back those who are recovering to using again.
  • Other papers also acknowledge that there is very little known about what kind of online Peer Services can be most helpful and therefore more research in this field is required
  • Some of these applications are more focused on a moderated chat between peers and mentors while others also offer modules and training courses to convey information about symptom management, path to recovery, and accurate medical advice.
  • PEER SURE App should have detailed guidance documentation about the specific services it is offering.
  • Additionally a list of expected features and functionalities of the app will be helpful to focus the energy on research and development towards the most important features.
  • Users of other Peer Support apps have recommended the peer support application should monitor for:
  • Talks for recurrence of drug use
  • Medical advice
  • Language that indicates racism, homophobia, and cyber victimization, and bullying
  • There is a lack of research on Peer Support applications, particularly on those that integrate NLP features. Though I found a research paper that focused on sentiment analysis to flag specific contents in text messages in peer support apps. However, I could not find any paper that mentioned the use of generative AI in Peer Support applications.
  • Generative AI can improve many existing application by making courses and modules more interactive (similar to how Khan Academy and Data Camp have implemented generative AI to help with the completion of courses on their respective platforms)
  • Additionally the use of generative AI could be helpful to provide company and chat assistance to patients at times when a Peer Mentor is not present. Given that isolation and lack of timely access to human specialists are major catalysts for the recurrence of drug use, perhaps there should be a feature for users to be able to access the chat bot on autopilot. This will allow users to connect with a chatbot at any time during the day or night (though this is just my idea. Dr.Madiraju has clarified a human should always be in the).
  • Generative AI can perform sentiment analysis and help with text moderation.
  • I hope these insights will assist with the development of the application and the fine-tuning of the language models for this project

Friday (6/9)

• Completed ChatGPT Prompt Engineering for Developers course: https://www.deeplearning.ai/short-courses/chatgpt-prompt-engineering-for-developers/
• Began watching videos on C#

2023 Week 3: (June 12th 2023-June 16th 2023)

Monday (6/12)

• Completed daily journal on Marquette Wiki
• Began Building Systems with the ChatGPT API course: https://www.deeplearning.ai/short-courses/building-systems-with-chatgpt/
• Continued watching videos on C#
• Practised training ChatGPT Prompt Engineering for Developers course content
• In future days, I will be meeting with Dr.Madiraju to decide if I should be focusing on practicing with Open AI API or developing with C# at this moment

Tuesday (6/13)

• Contacted Sajjad and discussed how to fine-tune an LLM model. He suggested a few sources as well as this paper: https://towardsdatascience.com/specialized-llms-chatgpt-lamda-galactica-codex-sparrow-and-more-ccccdd9f666f
• Contacted Christian and got an update on his progress
• Read more on moderation API from open AI. Given that our model will be interacting with OUD patients, it is important that we can test our model's semantic performance, even though we have licensed human mentors in the loop: https://platform.openai.com/docs/guides/moderation/overview

Wednesday (6/14)

• Attended group meeting and learned more about upcoming mini-presentations
• Met with Sajjad, Christian, and Dr. Madiraju to discuss the project progress:
  • Sajjad recommended that we switch the programming language for the web application from C# to Django

Thursday(6/15)

• Open AI offers pre-trained models without instruction which can be fine-tuned with a list of prompts and completion responses to those prompts in jsonl for example:

{

   	"prompt": "What is the capital of France?->",
   	"completion": """ The capital of France is Paris.\n"""

}, {

   	"prompt": "What is the primary function of the heart?->",
   	"completion": """ The primary function of the heart is to pump blood throughout the body.\n"""

}, {

   	"prompt": "What is photosynthesis?->",
   	"completion": """ Photosynthesis is the process by which green plants and some other organisms convert sunlight into chemical energy stored in the form of glucose.\n"""

},

• I have reviewed the list of available based models for fine-tuning. The following is based on information available from https://platform.openai.com/docs/models/gpt-4
• Given that the generative AI application will generate short responses to user questions, I have chosen curie based model for our project.

| Model | Speed | Performance | Price | Used best for | | --- | --- | --- | --- | --- | | Ada | Fastest | Good for simple tasks and short texts | Lowest | Text classification, sentiment analysis, summarization | | Babbage | Fast | Good for moderate tasks and longer texts | Low | Text generation, question answering, translation | | Curie | Moderate | Good for complex tasks and diverse domains | Moderate | Natural language understanding, dialogue, content creation | | Davinci | Slow | Best for any task and any domain | Highest | Semantic search, reasoning, creativity |

• Searched for tutorials for fine-tuning a base model with open AI API and tried replicating their code.

Friday(6/16)

• According to the Open AI fine-tuning page (https://platform.openai.com/docs/guides/fine-tuning), usually a few hundred examples are recommended for best results.
• Given that the generative AI application in PEER Sure app will draft a peer mentor's response to a user's question, I need to find a list of Q&As in opioid-related forums and scrap for user questions and peer mentor responses and later convert them to a JSON dictionary where questions are prompts and peer mentor responses are completions.
• When searching for Peer mentor conversations or mentee mentor Q/As on Google, I only found interview questions for those wishing to be licensed mentors and questions mentees should ask their mentors.
• There are other websites that offer mentorship services for those struggling with OUD, however, none of the reviewed sites offer public data on mentor/mentee conversations or questions.
• Next week, I will spend more time finding a way to curate prompts and completions for fine-tuning the job as well as reading new papers to better familiarize myself with online peer mentor services and studies, hoping to learn more about their data collection methods

2023 Week 4: (June 19th 2023-June 23rd 2023)

Monday (6/19)

• Read a few new papers. My focus with this batch of papers was online peer mentoring applications that preferably used generative AI or some form of NLP application.
• There are applications such as Koko (https://www.kokocares.org/) where they use generative AI along with a host of other peer mentoring services for mental health-related issues. However, there is no peer-reviewed paper associated with these websites, and therefore they are not useful for our research. (though if I had more time, I would sign up on one of these websites to get a sense of the user exprinece.)

• Development of a peer support mobile app and web-based lesson for adolescent mental health (Mind Your Mate)
  • Birrell, L., Furneaux-Bate, A., Debenham, J., Spallek, S., Newton, N., & Chapman, C. (2022). Development of a peer support mobile app and web-based lesson for adolescent mental health (Mind Your Mate): User-centered design approach. JMIR Formative Research, 6(5), e36068. https://doi.org/10.2196/36068
  • This paper describes the co-design process of Mind Your Mate, a program that consists of a smartphone app and an introductory classroom lesson to enhance peer support around anxiety, depression, and substance use for adolescents. The authors conducted a scoping review of existing peer support apps, a web-based survey and focus groups with 23 adolescents, and app development and beta-testing with experts and end users. The resulting program aims to empower adolescents to access evidence-based information and tools to better support peers regarding mental health and substance use–related issues. The paper also discusses the potential benefits and challenges of using digital technologies and mobile interventions for prevention initiatives targeting youth mental health and substance use.
  • Features of Mind Your Mate App:
  • Mind your Mate is a mobile app and web-based lesson that aims to help adolescents support their peers regarding mental health and substance use. The app provides evidence-based information, tips, tools, and resources on topics such as anxiety, depression, stress, alcohol and drugs, active listening, communication skills, harm minimization, and self-care. The app also allows users to create profiles for their friends, plan and follow up on conversations, track their moods, and customize their settings and appearance. The app was developed through a user-centered design process involving adolescents, experts, teachers, and counselors. The web-based lesson introduces the core concepts of the app and prompts students to download and explore it. Mind Your Mate is designed to empower young people to access peer support and improve their well-being.

• Experiences of using the digital support tool MeeToo: Mixed methods study. JMIR Pediatrics and Parenting (Tellmi)
  • Ravaccia, G. G., Johnson, S. L., Morgan, N., Lereya, S. T., & Edbrooke-Childs, J. (2022). Experiences of using the digital support tool MeeToo: Mixed methods study. JMIR Pediatrics and Parenting, 5(4), e37424. https://doi.org/10.2196/37424
  • This mixed methods study examined the impact and processes of a digital peer support tool called MeeToo (now Tellmi) for young people aged 11-25 years. The authors analyzed secondary data from feedback questionnaires completed by 876 young people at two-time points and primary data from semistructured interviews with 10 young people. The results showed that using MeeToo had positive impacts on young people’s mental health and well-being, such as making it easier to talk about difficult things, providing new ways to help oneself, feeling better, and feeling less alone. The authors identified two themes from the interviews that explained why using MeeToo had these impacts: anonymity and the MeeToo sense of community. Anonymity helped create a safe space for young people to express themselves freely and connect with others who had similar experiences. The MeeToo sense of community fostered a sense of social connectedness and support, which was especially valued during the COVID-19 pandemic. The authors concluded that MeeToo showed potential as a digital peer support tool for young people and suggested future research to examine how to sustain its impacts and processes. This study contributes to the literature on digital peer support for young people’s mental health by providing evidence of its effectiveness and mechanisms during a period of increased social isolation and stress due to COVID-19.

Tuesday (6/20)

• Met with Sajjad and discussed my progress thus far.
• Read another paper:
• Human–AI collaboration enables more empathic conversations in text-based peer-to-peer mental health support (HAILEY)
  • Sharma, A., Lin, I. W., Miner, A. S., Atkins, D. C., & Althoff, T. (2023). Human–AI collaboration enables more empathic conversations in text-based peer-to-peer mental health support. Nature Machine Intelligence, 5, 46–57. https://doi.org/10.1038/s42256-022-00593-2
  • This paper investigates how artificial intelligence (AI) can collaborate with humans to facilitate empathy in online peer-to-peer mental health support. The authors develop HAILEY, an AI-in-the-loop agent that provides just-in-time feedback to help peer supporters respond more empathically to support seekers. They evaluate HAILEY in a randomized controlled trial with 300 peer supporters from TalkLife, a large online peer-to-peer support platform. They find that HAILEY leads to a 19.6% increase in conversational empathy between peers overall and a 38.9% increase for peer supporters who self-identify as experiencing difficulty providing support. They also analyze the human-AI collaboration patterns and find that peer supporters use the AI feedback both directly and indirectly without becoming overly reliant on AI while reporting improved self-efficacy post-feedback. The paper demonstrates the potential of feedback-driven, AI-in-the-loop writing systems to empower humans in open-ended, social, and high-stakes tasks such as empathic conversations.

Wednesday (6/21)

• Attended a meeting with other cohorts and discussed the expectations for the upcoming presentation
• Met with Sajjad, Christian, and Dr. Madiraju to discuss the project progress
  • I discussed some of my ideas for next week's presentation and inquired about what would Dr.Maidraju expect from my slides
  • Sajjad suggested adding a slide to show what the project pipeline looks like.
  • They also suggested a few sources from which I can scrap data to imitate mentor/mentee question responses.

Thursday (6/22)

• I spent most of the day working on my presentation slides.
• The main slides for my presentation will focus on:
  • Start with basic information about the Opioid problem in Milwaukee as the motivation to create Peer Mentor Support APP (motivation and background)
  • What is unique about our approach
  • What has been accomplished so far (annotated bibliography: papers reviewed, data for training the model gathered, a demo of Chatbot has been made)
  • What Obstacles have I been facing
  • What are my future plans for this REU?

Friday (6/23)

• I received an email from Dr. Madiraju in which he shared the Open AI account information. Now I can experiment with my code.
• Continued working on my presentation slides

2023 Week 5: (June 26th 2023-June 30th 2023)

Monday (6/26)

• Reviewed Open AI API refrence:https://platform.openai.com/docs/api-reference/authentication
• After reviewing a few websites, I finally settled down for the Reddit Opioid recovery page where patients struggling with OUD can ask questions and others can respond in the comment section: https://www.reddit.com/r/OpiatesRecovery/
• The upside of using the Opoiod Recovery page is that all questions and responses (comments) are public. The theme and the topic of the questions are comparable with the kinds of questions we would expect our app users to ask
• Not all posts are relevant questions and even good questions can have irrelevant and harmful comments posted under them. My goal is to comb through the posts and select the relevant questions which our users may ask as prompts and the best comments for those questions as completions.
• My goal is to select relevant questions from the posts that our users may ask as prompts and the best comments for those questions as completions. I limit the length of the responses to 90 words (about 100 tokens) to save costs and provide brief answers. I use chat-gpt 4 to summarize longer comments if needed. This way, the model can learn from different OUD-related questions with various lengths and topics.*
• This process may take a while. Afterward, I will convert my dataset to jsonl lines for the fine-tuning task.

Tuesday (6/27)

• Met with Sajjad to share my progress.
• My goal is to have 100 examples for the training set and 30 examples for the validation set.
• Signed up for GPT-4 API access. Though it will not be accessible in time for this summer, it could still be helpful for the future of the PEER Sure App.

Wednesday (6/28)

• • Met with Sajjad, Christian, and Dr. Madiraju to discuss the project progress
• Attended a meeting with other cohorts.
• Today was a bit special given that I presented on my research progress for the first time.
• It was also nice to hear what others were working on
• Continued web scrapping from the Reddit page

Thursday(6/29)

• Continued web scrapping...

Friday(6/30)

• Benefits of model evaluation parts 1 and 2 → reducing hallucinations, wrong answers
• Two ways to Improve answers :

1. Providing sample question/ expert answers 2. Provide a Rubric (I am not sure if this will be necessary, but just in case)

• • If the above steps (especially step 1) are well executed, then we may not need a very large dataset to train our model: get a specific estimate of the number of interaction examples required (100 examples for training set/ 30 examples for validation set)

2023 Week 6: (July 3rd 2023-July 7th 2023)

Monday (7/3)

• Met with Sajjad and showed my demo chatbot as well as the issues I was having with my code

Tuesday (7/4)

• Continued web scraping and formatting my jsonl lines
• I recommend this website for validating your json lines: https://jsonlint.com/
• I recommend this website for converting your json lines to jsonl (the required formatting for Open AI API)

Wednesday (7/5)

• Met with Sajjad, Christian, and Dr. Madiraju to discuss the project progress
  • Continued web scraping and formatting my jsonl lines

Thursday(7/6)

• Taking the day off to prepare for my trip to Pittsburgh

Friday(7/7)

• My training and validation datasets are almost ready.
• I have tried to replicate a few tutorials for fine-tuning my model. However the most useful of all, thus far, has been the Data Camp tutorial for fine-tuning GPT-3 in Python: https://www.datacamp.com/tutorial/fine-tuning-gpt-3-using-the-open-ai-api-and-python