Since mobile apps have implemented performance metrics since Q1 2024, we can measure and identify what kind of error status code or average response time on each feature that was used by users. The median response time when a user applies for a job is within 5 seconds for both iOS and Android, i think this is 2-3 seconds slower than when we apply for a job using web-based. I wonder what makes the response time when applying for a job in the app take more than 3 seconds, i hypothesize that several conditions cause something like that such as:
- after applying for a job we also bring up a bulk application option, whereas this bulk application is related to the job title that the user applied for previously (need to investigate furthermore and prevent any kind of issue that might occur)
Personally speaking, this gave me a flashback back then on 2021, when we had a major incident on mobile apps from 18-21 August, there were at least 883 users who couldn’t apply for any job because of invalid data and the user got timeout.
the data was retrieved from Firebase app performance
Another Learning
Somehow, when we were trying to increase the test case coverage during regression test it seems to create more robust testing and result in a better test case pass rate. However, when i tried to compare it with the actual issue that was found on the production environment, the test case pass rate seemed meaningless due to the effectiveness below 70 %. I calculated based on the last 3 versions of mobile apps released, most of them “just” reach 50 %. By all means: that if you find 2 issues during development and this issue has been fixed, there is a chance of 50 % that 2 different issues will occurs on production.
Again, I need to refine and readjust my testing point of view to this team