After using generative AI for the whole year on my daily work, i can conclude that:
- it oftens hallucinated and generated out of context
- sometimes it’s helpful for quick prototyping or analyzing what went wrong
- but sometimes it’s quite dreadful and causes generating flaky tests more than ever
- sometimes it’s boost my productivity, but the other day, it will goes hampered my productivity
and all of these are mostly caused by only 1 problem: the context that I’ve provided is not detail or comprehensive
take for example, if the ticket is being written half-assed or half-baked, 100% of all these AI tools only provide and generate their response also half-baked. The generated responses from Claude are even worse. they will start ‘mumbling around’ and generate at least 10-20 test cases which majority of them weren’t testable or executable due to the lack of context and resulting in not suitable response
and guess what? it’s all goes back again to me, manually evaluating, writing additional information, given the AI more context and prompting again → causing only perform back and forth effort
In essence: AI doesn’t replace careful, context-rich specification. It amplifies whatever i give to them. when the specs or requirements are vague, AI accelerates the vagueness. otherwise, when the specs are sharp or comprehensive, AI can accelerate my productivity
in the end, it’s not just about which model is better or consistent, which prompts or base knowledge that needs to be determined so it can generate a ‘woah’ result, but it’s all goes that we need tight or rigid requirement / specs before passed to the AI