Hong Kong chatbot still a work in progress

Key Concepts

• HK Chat: An AI chatbot application launched on the Hong Kong IM Smart App.
• HKGAI (Hong Kong Generative AI Research and Development Center): The developer of HK Chat.
• Beta Testing: The current stage of development for HK Chat, emphasizing user feedback and refinement.
• Accuracy vs. Speed: A stated prioritization by HKGAI, favoring accurate responses even at the cost of slower processing times.
• Hong Kong Observatory: The official source for weather information in Hong Kong, used for comparison in the testing.
• MTR Monthly Pass: A transportation pass offering unlimited travel between specified stations for a fixed monthly fee.

Introduction of HK Chat and its Purpose

HK Chat, an AI chatbot developed by the Hong Kong Generative AI Research and Development Center (HKGAI), has been launched on the Hong Kong IM Smart App. The app is designed to provide instant answers to questions about Hong Kong, covering a broad range of topics including weather, transportation, food, housing, stocks, and even legal advice. The stated goal is to serve as a readily accessible information resource for residents and visitors.

Performance and Response Times

Initial testing reveals significant performance issues, specifically slow response times. A simple query regarding directions to Victoria Peak took approximately one minute to resolve. HKGAI acknowledges this, stating that the development team intentionally prioritizes “accuracy and quality over speed.” This suggests a deliberate design choice to ensure reliable information, even if it means a less responsive user experience.

Weather Forecast Accuracy – Discrepancies and Clarification

A test of the weather forecasting capabilities highlighted inaccuracies. When asked about the probability of rain in Trumun on December 19th, HK Chat initially predicted a 90% chance of precipitation. However, the Hong Kong Observatory, the official meteorological source, reported only a 20% chance.

HKGAI subsequently directed testers to a dedicated “weather tab” within the app, claiming it draws information directly from the Hong Kong Observatory. This tab provided a more accurate forecast, but with considerably less detailed information compared to the Observatory’s full report. This indicates a potential issue with the initial data source or processing method used by HK Chat.

Transportation Planning – Errors in Cost Calculation

Testing the app’s ability to handle transportation planning revealed further inaccuracies. A scenario was presented involving a commuter traveling from Tong Chong to Causeway Bay five days a week, and inquiring about potential savings from utilizing the Tong Chong MTR monthly pass (which provides unlimited travel between Tong Chong and Hong Kong station).

After a processing time of approximately three minutes, HK Chat provided incorrect information regarding both the price of the monthly pass and the standard adult fare for the journey. This demonstrates a failure to accurately calculate costs and provide practical transportation advice.

Beta Testing and Ongoing Development

HKGAI acknowledges that HK Chat is currently in its beta testing phase. The organization emphasizes that the app is undergoing continuous refinement to improve the accuracy of its responses. Developers are actively soliciting user feedback to aid in this process, recognizing the importance of real-world testing in identifying and correcting errors. As stated by HKGAI, the app is “very much a work in progress” based on the initial tests conducted on weather and transit planning.

Conclusion

The initial assessment of HK Chat reveals significant shortcomings in both response time and accuracy, particularly in areas requiring numerical calculations and access to real-time data. While the developers’ commitment to prioritizing accuracy is commendable, the current performance level suggests the app is not yet a reliable source of information. The ongoing beta testing phase and emphasis on user feedback are crucial steps towards improving the app’s functionality and fulfilling its intended purpose as a comprehensive information resource for Hong Kong.