Art of Reading
Student Innovation Project Brief

Revised 8/11/2021 by Daniel Silva

Technical Field

This project involves the technical fields of Programming, Augmented Reality, and Artificial Intelligence.

Background Information

The inspiration behind this project was birthed from my passion for learning how to learn, which evolved into a concept that could be used to teach my children how to read.  Coupled with my degree in Game Programming, I realized that I wanted to create an application that would facilitate a visual style of learning and engage children through interactive puzzles.  As an additional challenge, I chose to present this concept through augmented reality and incorporate artificial intelligence into the learning process.

Prior Art

Inception XR LTD has produced the augmented reality application “Bookful”, which has a focus on presenting a variety of children’s books via an augmented reality experience (BusinessWire, 2020).  “Bookful is a magical, playful universe for book lovers that uses augmented reality to bring books to life” (Bookful, 2020).  This application, as well as a few similar applications, utilize AR to support a specific library of stories.  My hope would be to create an application that will recognize words or objects and present a learning experience, not tell a story.

Shapes 3D – Geometry Learning” by Learn Teach Explore Sp. z o.o., is an augmented reality program that is designed to teach about the geometry of 3D objects (Apple, 2019).  This program allows the user to interact with various shapes to learn about the shape structure, faces, vertices, and edges.  Although Shapes 3D makes use of AR to facilitate teaching and learning, it is substantially different from Art of Reading in the material that is being presented.

Project Description

This project is innovative because it will utilize artificial intelligence to recognize words and everyday objects, and then process that recognition to create an augmented, spelling-based, puzzle.  The approach to functionality will be similar, in manner, to procedural map generation, where the AI, in Art of Reading, will be able to generate any word puzzle based on what it sees.

Innovation Claim

Art of Reading is a fun and interactive platform that will compel children to learn how to read and improve on spelling.  Using Augmented Reality and Artificial Intelligence, Art of Reading will analyze words or physical objects, through computer vision, and then present interactive learning puzzles, based on those objects, back to the user.  The focused target, for Art of Reading, will be children that are learning how to read and spell, and the teachers that are guiding them in that learning process.

Usage Scenario

This project was inspired by a concept, utilized in speed-reading, that encourages the mental visualization of ideas as images to effectively boost reading comprehension (Butler, 2014).  Not only is this program a useful tool for anyone that is just learning how to read and spell, but it would also facilitate object to word recognition for anyone attempting to learn the English language.  Finally, Art of Reading could be an invaluable resource for anyone struggling with learning or reading disabilities, such as dyslexia.

Evaluation Criteria

The following questions will identify the successful completion of this project:
- Can the application be used with any source of text?
           - Is the text material recognized regardless of the text font or size?
           - Can the application recognize and convert a hand-written word?

- Can the application be used with any object?
           
- Is the object recognized regardless of the surrounding environment?
           - Can objects, residing together, be isolated as individual elements?

- Has the word or object been properly recognized and translated?
           - Is the correct spelling puzzle being displayed to the user?

Objectives and Tasks

Goal: Develop an Augmented Reality application that can utilize artificial intelligence to convert words and everyday objects into interactive spelling puzzles.
Timeline: Currently enrolled in SIP409, with a total of 5 credit hours remaining for my degree completion.  This leaves about a month of development time for a finalized product.

Objective 1: Learn how to implement augmented reality using the Unity Engine.
           - Download and install Unity with Vuforia: TASK COMPLETE
           
- Purchase a Vuforia course from Udemy and start learning: TASK COMPLETE
           
Review YouTube videos, and alternate learning resources, for best implementation practices of AR, using Unity with Vuforia.

- Objective 2: Set up an Augmented Reality project, in Unity, and begin concept testing
           - Create a Unity project, utilizing Vuforia, and begin testing on AR implementation: TASK COMPLETE
           - Test the feasibility of mapping an image to a string of text: TASK COMPLETE
           - Test the feasibility of mapping a puzzle to an object: TASK COMPLETE

- Objective 3: Develop the game logic for a functional spelling puzzle, based on observed objects.
           -
Generate the proper letters for the observed item: TASK COMPLETE
           -
Create a gameboard that can recognize the proper sequence of letters: TASK COMPLETE
           - Implement game logic that will provide the user with feedback and continue to drive the game forward.


Objective 4: Learn about integrating Artificial Intelligence with Augmented Reality.
           - Discover existing frameworks that already implement machine learning with AR.
           - Research and learn about Unity's "Perception" toolkit.
           - Research and learn about Unity’s “Barracuda” project (Rothmann, 2019).

Description of Design Prototype

The design prototype for Art of Reading was created in the Unity Game Engine, utilizing Unity 3D with the Vuforia AR package.  Currently, the prototype works to present a conceptual view of what the program could accomplish in future iterations and is designed as a mobile application.  Art of Reading currently features several spelling puzzles, scattered letter blocks, and platform indicators for where the letters should be placed.  The prototype also utilizes multiple scripts to present the puzzle, reveal the solution animation, manage touch features, and handle the user interface.  When the designated object is detected, within the parameters of the AR camera, the associated puzzle will be revealed.   Upon the successful completion of a puzzle, a solution animation will be played as a reward.  Finally, Art of Reading will feature a menu option that will allow the user to reset or exit the program.

Evaluation Plan

Project Completion Assessment

Art of Reading will require an iterative approach to program development with intermittent testing throughout the process.  In order to properly evaluate the given criteria, Art of Reading will need to fully implement an augmented reality program that effectively utilizes artificial intelligence to convert machine vision into machine learning.  The program will need to recognize all forms of text, regardless of the font size or style, and will, eventually, even cross language barriers.  Art of Reading will also need to recognize common objects and generate spelling-based puzzles derived from either the text or the object.  Proper testing can be performed by myself, as well as by my two-year-old and three-year-old children, via the test mobile application.

What went right? – From the first iteration and initial direction of Art of Reading, there have been some significant changes to the entire program.  Art of Reading now has a more manageable end goal and a more engaging and interactive interface for children.  At its core, it still manages to utilize AR and AI to generate learning materials, with only a slight variation in the functional direction.  Vuforia has been relatively easy to pick up, and mapping images and objects is a quick and painless process.

What went wrong? – The initial direction for Art of Reading was a bit overreaching and exponentially more difficult to execute within the allotted time.  Although the new direction does show some promise, there have been issues with anchoring the puzzle scenes to just the object itself, requiring that an anchoring image also be incorporated.  Finally, concerning the integration of artificial intelligence, into this project, I have just learned about an AI training model, in Unity, called Perception.  Although I do not foresee adding full AI integration into my project, at this point, I will continue to practice with Perception and hopefully implement the AI portion in the near future.



Lessons learned: Project timeline and scope are crucial elements that should be fully worked out from the onset.  Changes may come and features may change, but it is essential to maintain focus on the overall objective.  Most importantly, managing every minute of your day, properly, is the only way to juggle a full-time degree, job, and family life.

What could be done differently? – This project started, for me, at the end of last year, and received bits of my attention between a variety of alternate obligations.  Had I heeded the warnings of my professors and peers, I would have put more thought into this project prior to the Student Innovation courses, and more time would have been dedicated, to development, in the space between the SIP courses.

Appendices

Appendix A: References
Apple. (2019). Shapes 3D - Geometry Learning. Retrieved July 1, 2021, from App Store Preview:
           https://apps.apple.com/ae/app/shapes-3d-geometry-learning/id501650786#?platform=ipad
Bookful. (2020). About Bookful. Retrieved November 29, 2020, from Bookful: https://bookful.app/about/
BusinessWire. (2020, January 21). Inception XR Launches Augmented Reality Kids Library Service in Partnership With LG U+. Retrieved November 29,                   2020, from Business Wire: https://www.businesswire.com/news/home/20200121006055/en/
Butler, D. (2014). Speed Reading with the Right Brain: Learn to Read Ideas Instead of Just Words. Createspace Independent Publishing Platform.
Rothmann, D. (2019, November 4). Enhancing AR with machine learning. Retrieved December 16, 2020, from towards data science:                                                     https://towardsdatascience.com/enhancing-ar-with-machine-learning-9214d2da75a6

 

Figure 1: Art of Reading: Anchor Image

Follow me on Social Media