Difference between revisions of "User:Grberlstein"
From REU@MU
Grberlstein (Talk | contribs) |
Grberlstein (Talk | contribs) |
||
Line 143: | Line 143: | ||
*Wrote a rough draft of the literature review | *Wrote a rough draft of the literature review | ||
*Created the poster for the poster session | *Created the poster for the poster session | ||
+ | |||
+ | =='''Week Ten (7/31 - 8/4)'''== | ||
+ | *Gave poster presentation | ||
+ | *Gave REU project presentation | ||
+ | *Reconfigured the maps to work with the other half of the data set | ||
+ | *Created more graphics for the paper | ||
+ | *Wrote a (very) rough draft of the discussion section | ||
+ | *Read a few more sources for the paper | ||
+ | *Minor tweaks to the map's color palette | ||
+ | *Departed for home |
Latest revision as of 16:27, 7 August 2017
Contents
Griffin Berlstein
Griffin is an incoming junior majoring in Mathematics and Computer Science at Vassar College in Poughkeepsie, New York.
Readings
Background
Algorithmic Ethics
- Ethics of Algorithms
- Is There an Ethics of Algorithms?
- Toward an Ethics of Algorithms
- Quantifying Search Bias
- Understanding and Designing around Users' Interaction with Hidden Algorithms in Sociotechnical Systems
- The ethics of algorithms: Mapping the debate
Clustering and Data Science
- Algorithms for Clustering Data
- K-means Clustering in Python
- Critique and Contribute: A Practice-Based Framework for Improving Critical Data Studies and Data Science
Project Log For Summer 2017
Week One (5/30 - 6/2)
Day 1 (5/30)
- Attended REU orientation
- Obtained ID card and computer access
- Met with Dr. Guha and discussed broad ideas surrounding the project
Day 2 (5/31)
- Attended Library orientation
- Finished reading Ethics of Algorithms by Thijs Slot. This was the last of the pre-REU reading.
- Started reviewing the basics of Python
- Given crime data sets to review by Dr. Guha
Day 3 (6/1)
- Attended a meeting on proper research practices by Dr. Factor
- Set up direct deposit
- Reviewed the basics of GitHub
- Continued to review Python
- Examined crime data and the various ways it was made publically available
Day 4 (6/2)
- Moved mentor meeting to Wednesday due to scheduling issue
- Started reading background information provided by Dr. Guha
- Set up Jupyter notebook and the various dependent libraries
- Created rough implementation of K-means clustering on random data
- Obtained card access to Dr. Guha's lab
- Posted rough, pre-discussion milestones
Week Two (6/5 - 6/9)
Day 1 (6/5)
- Refined K-means implementation with the K-means++ seeding described in the Data Science Lab article
- Tested the algorithm on random Gaussian distributions, rather than random points
- Experimented with visual plotting of the algorithm using Seaborn and Matplotlib
Day 2 (6/6)
- Attended RCR training
- Finished reading the relevant sections of Algorithms for Clustering Data
- Experimented with Scikit-learn's implementation of K-means
Day 3 (6/7)
- Met with Dr. Guha and discussed the immediate future
- Set the goal to produce an interactive crime map by next Wednesday
- Gathered data from website and began sorting
Day 4 (6/8)
- Created a script to aggregate the data from multiple spreadsheets into a single usable file
- Looked into potential libraries needed to create the interactive map
- Ran into issues with the format of the data location
- Converted the addresses in the data into latitude/longitude coordinates
Day 5 (6/9)
- Found a publically available shape file of the city
- Set up the necessary scripts to display the file
- Ran into an issue with the points not being in the same coordinate system as the shape file
Week Three (6/12 - 6/16)
Day 1 (6/12)
- Fixed point plotting to align with shapefile
- Added choropleth coloring by neighborhood
- Started reading The ethics of algorithms: Mapping the debate
Day 2 (6/13)
- Finished The ethics of algorithms: Mapping the debate
- Started reading Weapons of Math Destruction
- Started implementation of website from GitHub
- Established the needed dependencies to run a local instance of Jekyll
Day 3 (6/14)
- Finished website framework
- Uploaded initial map version
- Started on the second version of the map
Day 4 (6/14)
- Split the data into multiple sets
- Used K-Means to sort in a variety of ways
- Wrote a python script to run K-Means multiple times and output results to be fed into D3
Day 5 (6/15)
- Put modified data into D3 setup for the new map
- Tweaked basic settings
- Added ability to display different variations of K-Means on the map
Week Four (6/19 - 6/23)
Day 1 (6/19)
- Tweaked the map visuals
- Added a convex hull to display the cluster borders
- Made the convex hull creation dynamic and attached to the data, rather than precomputed in the data frame
Day 2 (6/20)
- Added more visual tweaks to the map
- Added a grid to the display and fixed inaccurate axis labels
- Evaluated the relevancy and accuracy of the different clusters produced
Day 3 (6/21)
- Compared Milwaukee crime reports against produced clusters to gauge accuracy
- Read Critique and Contribute: A Practice-Based Framework for Improving Critical Data Studies and Data Science
- Met with Dr. Guha and discussed the next step in the project
Day 4 (6/22)
- Started programming a (mostly) vectorized implementation of K-Means to later modify
- Continued reading Weapons of Math Destruction
- Had the weekly working lunch and began early outlines of the mini-presentations
Day 5 (6/23)
- Fixed vectorized implementation of K-Means
- Tested implementation on random datasets and compared with the results of Sci-Kit Learn's implementation
- Implemented a geodesic distance metric using the Haversine great circle distance formula
- Modified my implementation of K-Means to use the new distance metric and build a test framework to compare clustering with the geodesic distance and Euclidean distance from the same set of starting points.
Week Five (6/26 - 6/30)
- Tested side-by-side visualizations for geodesic vs Euclidean clusterings
- Experimented with multiple methods of visualizations
- Finished reading Weapons of Math Destruction
- Gave mini-presentation
- Ran into difficulties with the public datasets on the Milwaukee website
Week Six (7/3 - 7/7)
- Got better datasets and resolved issues with publically available census data
- Overlayed demographic information on the maps
- Generated a demographic breakdown for each cluster and compared geodesic vs euclidean
- Merged functionality from different versions of the map
Weeks Seven to Nine (7/10 - 7/28)
- Failed to do logs consistently
- Finalized demographic overlay
- Implemented multiple version of a potential bias index
- Ran experiments on the data to get trends about the potential bias index
- Expanded data set to include all available years worth of data
- Geocoded all of the new data
- Expanded map functionality to include potential bias index and cluster similarity
- Added interactive graphs for demographics and potential bias
- Moved potential bias calculations to Python to allow for faster web access
- Read lots of papers for the literature review
- Wrote a rough draft of the literature review
- Created the poster for the poster session
Week Ten (7/31 - 8/4)
- Gave poster presentation
- Gave REU project presentation
- Reconfigured the maps to work with the other half of the data set
- Created more graphics for the paper
- Wrote a (very) rough draft of the discussion section
- Read a few more sources for the paper
- Minor tweaks to the map's color palette
- Departed for home