User:Matthew

Tuesday, May 30

• met new people
• heard some people talk
• ate some panera
• heard more talking
• Started to get up to speed with TA Bot project.
• In particular, I read a paper (``Experiences with TA-Bot in CS1") highlighting experiences with using TA Bot for Marquette CS classes, and read a survey that was given out to assess the effectiveness of the TA Bot.
  • starting to understand the leveling system: it only concerns test cases, and the levels represent difficulty of the tests (higher levels corner/edge cases for example)
  • big emphasis on the TBS system to encourage students to start work early, but many did not like it
  • how are the learning outcomes affected? how do we measure good learning outcomes
• Also, got started with looking at the database of submission scores from TA Bot, and am beginning to look at Python libraries that will help me manipulate this data.
  • using base code as a reference
  • pandas library: I understand the very basics of series and data frames
  • need to understand sorting/grouping/splitting

May 31, 2023

• heard some talking
• getting comfortable with `pandas` and getting pertinent parts of the TA Bot submissions database
• brainstorming ideas for comparisons/visualizations we want between TBS and non TBS semesters to assess positive/negative student outcomes
  • right now, we focus on the effects on linter errors: how much linter errors go down using TBS vs no TBS, and if students correct linter errors even after attaining 100% on an assignment
• made a graph comparing the average reduction in linter errors from a student's first submission to his last per assignment with TBS vs. no TBS
  • a clear correlation in assignments 1-5 that showed that TBS had a higher reduction in linter errors
  • assignments 6-10 are not so clear. Brylow: either students aren't making as many errors or they are just not correcting them
• some other data gathered, needing visualizations
  • students submit far fewer times on average using TBS for a given assignment
  • students tend to resubmit more often after reaching 100% without TBS though the numbers are both low
  • we also studied the number of linter error reductions after reaching 100% w//w/o TBS, but the data does not make entirely clear any overarching trends (that might also help explain the 1-5/6-10 disparity)

1 June 2023

• refactored visualization code
• made visualizations of data from two more semesters
  • those semesters did not use TBS
  • improvements not very obvious, but the two new semesters did not use the same projects, so other factors may be at play
• thought of idea for new visualizations
  • looking at percent change of linter errors reduced instead of just the number reduced
  • instead of comparing to a student's first submission (may be a test or a mess, which is unreliable) look at submissions beyond a certain scoring threshold (like 70%)
• meeting with Dr. Islam
• read the following papers studying failure rates of introductory CS courses:
  • ``My Program is Correct But it Doesn’t Run: A Preliminary Investigation of Novice Programmers’ Problems"
  • ``Failure Rates in Introductory Programming Revisited"
  • ``Pass Rates in Introductory Programming and in other STEM Disciplines"
  • ``Failure Rates in Introductory Programming — 12 Years Later"

Friday, June 2 2023

• created new visualizations comparing reductions in pylint errors between submissions that score 70% or more, and submissions that are passing
  • clear data that suggests TBS is helping reduce more pylint errors
  • further work needs to be studied on the later assignments: is TBS helping students to create fewer linter errors in the later semesters (so that they wouldn't have many to fix)
• talked to Dr. Brylow about stuff
  • without TBS there was also no grade for linters (so do students really reduce linter errors when they pass all correctness tests?)
  • need to move on from looking at just averages and start looking at measures of spread and outliers in linter numbers
  • also got many tips on writing the paper and telling a story about the data with the visualizations
  • who are these overachievers?
• made pie graphs representing percentage of students with passing submissions who resubmitted
  • total is about 20%. need further analysis on who these people are
  • comparing number of linter errors with people who submitted only once vs multiple times
• worked on visualizations regarding students reducing pylint errors even after getting all the test cases, comparing this to students who did not resubmit
  • in Fall 2021: students who did not resubmit had lower number of average linter errors than students who passed (comparing to their first passing submission), and the students later resubmitted lowering the number of linter errors to comparable numbers to the non-resubmitters
  • in Spring 2022: no large trends showing that resubmitters resubmitted to lower the number of pylint errors they had; values stayed the same as the nonresubmitters (and still remain larger than TBS semester)
• RCR training

6/5/2023

• RCR talk with Brylow
• read paper "Investigating Static Analysis Errors in Student Java Programs" in preparation for presentation the next day

Tuesday 6/5

• created line graph of number of pylint errors on average per day before the due date
  • no useful information gained
• met with Dr. Islam
  • talking about paper
  • looking at the data more and seeing new patterns emerge
  • reading papers in the last 5 years from SIGCSE