It is usually necessary to make sure that the benchmark atmosphere is just like the enterprise manufacturing atmosphere, he mentioned, and to doc areas the place community, compute, storage, inputs, outputs, and contextual augmentation of the benchmark atmosphere differ from the manufacturing atmosphere.
Additional, make it possible for the mannequin examined matches the mannequin that’s accessible for preview or for manufacturing, Park suggested. It’s common for fashions to be optimized for a benchmark, with out revealing deep element into the associated fee or time required for the coaching, augmentation, or tuning going into that optimization.
In the end, “companies in search of to conduct a aggressive analysis of AI fashions can use benchmarks as a place to begin, however actually need to state of affairs take a look at in their very own company or cloud environments if they need an correct understanding of how a mannequin may match for them,” Park emphasised.
