The company aims to improve ELMAR’s speed, accuracy and cost-effectiveness for training, with plans to scale up the model post-beta cycle.
#BENCHMARK AI XWING PDF#
The ELMAR model has been put through its paces against several knowledge bases such as Zendesk and Confluence, as well as large-sized PDF documents.įollowing successful alpha feedback, Got It AI plans to soon commence ELMAR’s beta program with enterprise pilots across multiple industries and receive feedback on the types of pre-processing and post-processing “alignment” that work across all industries, versus those that are industry or enterprise-specific. “So the enterprise user sets its policies for removing data, such as personally identifiable information (PII).” “The pre-processor will be tuned, configured and controlled by the enterprise,” Relan told VentureBeat.
Got It AI’s ELMAR language model allows businesses to configure their pre-processors and plan measures to secure their language model architecture against attacks. We had to thread the needle and then find the right combination of a commercializable model, training techniques and data.” Empowering businesses with greater LLM control “We picked our approach to be such that ELMAR’s model, training and data are not constrained by the licenses of LLaMA and Alpaca-like models and data,” said Relan. Got It AI’s hallucination rate comparison. On the other hand, ELMAR, when fine-tuned on the same dataset, produced accurate results, equivalent to ChatGPT-3,” said Khatri. “When we used Alpaca, an open-source model, for a Q&A task on our target 100 articles set, it resulted in a significant fraction of answers being incorrect or hallucinations, but did better after fine-tuning. Got It AI’s study revealed that smaller open-source LLMs perform poorly on specific tasks unless they are fine-tuned on target datasets. “It is not just about the data, but also about modern model architectures and training techniques.”Įarlier in January, the company developed what they called “ TruthChecker,” a small language model–based fine-tuned post-processor, which compares responses generated by any language model with ground truth in the target dataset and flags what appear to be incorrect, misleading or incomplete answers a phenomenon known as “hallucination.” Despite fine-tuning, such models performed significantly worse than other more advanced models,” said Chandra Khatri, head of conversational AI research and cofounder of Got It AI. In our experiments, we did not find this to be the case. “Recently, it was suggested that smaller and older models like GPT-J can deliver ChatGPT-like experiences.
The study demonstrated how a smaller yet fine-tuned LLM can perform just as well on dialog-based use cases on a 100-article test set made available now for beta testers. To advance conversation surrounding the accuracy of language models, Got It AI compared ELMAR to OpenAI’s ChatGPT, GPT-3, GPT-4, GPT-J/Dolly, Meta’s LLaMA, and Stanford’s Alpaca in a study to measure hallucination rates. “We are saying all that power is not necessary for key enterprise use cases and requirements.” Image Source: Got It AI “We are not saying very powerful models aren’t needed,” Relan told VentureBeat. Furthermore, ELMAR allows for fine-tuning on the target dataset, eliminating the need for costly API-based models and preventing a surge in inference costs. Firstly, due to its diminutive size, the hardware required to operate ELMAR is significantly less expensive than that needed for OpenAI’s GPT-4. Got It AI claims that ELMAR offers several benefits to enterprises seeking to incorporate a language model.
0 kommentar(er)
