2022-10

TUM.ai Makeathon: Skin Cancer Detection App

I joined the TUM.ai makeathon without much thought or preperation. I just wanted to make something cool and eat free pizza, and I think I did (both)! We made an app that can use the smartphone camera to detect melanoma (deadliest type of skin cancer) at an early stage, since it's very easily treatable at that point (just cut it off and put a bandade over it), but most deaths result from it being overlooked by doctors until it's too late.

For the 48h makeathon, I managed to assemble a team after lots of chaos, and originally convinced them to make an extension for social media, let's say twitter, where it would use NLP to try to analyse a post's opinion on a certain topic and try to recommend you a broad selection of posts with explicitly different opinions and perspectives. The idea was that social media tends to put people in a bubble or an echo chamber, where their opinions get more and more extreme; therefore showing a much more broader and balanced set of ideas that much more accurately represents the spectrum/consensus of the public might help combat toxicity on the internet, and in turn, in society.
But then after the actual challenges were publicized, we decided on the melanoma detection instead, since it looked more impactful and interesting. I mostly took care of the Flutter front-end, and the python backend for the inference using a PyTorch model, both of which I didn't have too much experience with. So I had to basically learn everything on-the-fly which resulted in me working for around 30h until the end of the deadline and pitch. I actually quite like Flutter and found it pretty easy, since Dart is very Java-like with a lot of nice modern syntactic sugar sprinkled on top, and the UI framework is very polished.
Anyways being my first makeathon/hackathon, I had a lot of fun and am proud of everything we learned and pulled off. It was very much worth the lack of sleep! There could have been more free pizza though, a lot of us were kind of starving...
We named it MelaScan and the goal of the app was simplicity and minimizing the time required from tapping the app icon until the analysis result is shown. For more details, see our embedded pitch slides below or visit the project on DevPost.

Screenshots:
Home screen Scanning Negative result Positive result
Our Final Pitch Slides: (direct link)


2022-SS

TUM: Augmented Reality Project

I always refused to learn to read the piano sheets as a kid because it was so exhausting to me and felt like there must be an easier way. So I made one (with my friends).

For the Augmented Reality lecture at TUM, we could earn a grade bonus by working on a 1-month group project for a basic AR application. Originally I wanted to make Line RidAR, a pen-and-paper AR version of Line Rider where you would point a camera at a paper (or whiteboard or tablet, or really any drawable surface) and draw the track lines on it. Then you could tap play on the device and the sled rider would be spawned and you get to watch it slide down the lines you drew.
While it would have been a neat little tech demo, I chose it mostly to keep the scope of the project small because my semester was already very busy. So we decided to go for a project that is the total opposite! I probably sunk at least 100 hours into this... :')
It works on the Oculus Quest 2 headset, using the Passthrough API, making it a pseudo-AR experience. All you have to do is press the keys according to the falling tiles, like in a lot of rhythm games. You, of course, still need a real piano though technically, with really good handtracking we could probably do full virtual (barely) playble piano that would suck compared to a real piano but still kinda cool regardless.
It was my first time diving (somewhat) deep into modern OpenGL and doing anything with OpenXR, along with using Android Studio. It was fun and very rewarding, especially building the small engine that simplified most of the low-level stuff away into easy-to-use high-level classes. Of course it was often times frustrating as well, with the lack of good documentation and all the gradle issues...
It's fully open-source and you could install a pre-compiled APK onto your headset if you have one, see link below.

Short explanation:

In-class presentation: