Guidance
Technical Field Expert for Mechnical Systems
Inspiration
What inspired you to build this project?
A useful, effective, and easy solution for complex technical problems
What It Does
Describe the main features and how someone would use it.
It provides an easy way for technicians to fetch key information accurately. The framework promotes a conversational communication structure mimicking a foreman-apprentice based relationship.
How We Built It
Share the tools, APIs, frameworks, and design decisions behind your project.
We used AI coding, we used the gemini API for multimodal data processing. We created a sequential prompt pipeline based on the model restrictions and our requirements.
Challenges We Ran Into
What was difficult, surprising, or especially interesting to solve?
Merge conflicts for a fast-paced, small repo. Connecting web of APIs for several processes.
When recording our questions for the model to answer, we faced a roadblock where the microphone was on, we were speaking, but the web app wasn’t detecting any input. This problem was caused by a NotSupportedError on the MediaRecorder. This was happening because our React app was in Strict mode for testing, which means it was mounting and unmounting the component while the getUserMedia() function was still setting up the microphone. We added a flag inside the useEffect, so just in case the microphone reloads or closes the mic setup early, the old microphone setup gets shut down, ending the loop and allowing it to function properly. We learned that setting up live audio is tricky because permissions and setup happen at the same time.
When the AI was responding using TTS, it kept interrupting itself, as it thought the audio from the speakers was the user speaking again, stopping itself. We fixed this by implementing a voice detection loop, checking if the TTS was outputting and forcing the detected audio to a level 0 so there was no chance of the TTS triggering a recording. All while staying up for 24+ hours.
Accomplishments That We Are Proud Of
What are you most excited about?
We set up the server to use phones as an input, this allows the operator to easily submit images of many angles.
What We Learned
What did your team learn while building this?
We learned how to use the gemini API, better teamwork with detailed cooperation. We also learned the prompt created some funny images.
What Is Next
Where would you take this project after the hackathon?
We likely continue to develop the project, it has potential.