Reduce the time required to evaluate the performance of Text Summarization Models using Tiyaro Experiments

Text summarization is the task of condensing a huge block of text into a shorter concise version without changing the intended meaning. Due to an ever-increasing deluge of textual data like documents, articles, Electronic Health Records, legal documents, etc, it is important to condense such data into smaller concise chunks for faster analysis and consumption. There are a variety of use cases for text summarization across different industries for example companies producing long-form content, like whitepapers, e-books, and blogs might be able to leverage summarization to break down this content and make it sharable on social media sites like Twitter or Facebook. This would allow companies to further re-use existing content. Another common example is using summarization on a constant flow of emails, this could surface the most important content within an email and let us skim emails faster. Helpful for monitoring customer complaints as well as managing helpdesk at an organization.

To create a quick solution for one of such problems, we can quickly leverage Tiyaro's Platform to search among thousands of pre-trained Machine Learning models and SaaS APIs to find the one that best suits our use case. We can utilize Tiyaro's Experiments to compare the best-of-class text summarization models accelerating the time required to select a model from months to merely a few minutes.

Accelerating the rate of model evaluation using: Tiyaro Experiments

We evaluate the performance of text summarization models on an assorted set of news articles by using the Tiyaro Experiments . We selected the following pre-trained models for our text summarization experiment:

• Bart Large Xsum
• T5 Base CNN
• Pegasus Xsum
• Bart Base CNN A few things to note here are that the pre-trained models we selected are of varying complexities, have different parameters, and are trained on different datasets. As seen in the video above we start by searching the models by using Tiyaro Explore . Once we have completed searching and using demos for different models we can head on over to create experiments. There are various ways to create experiments but we are simply going over to the experiments tab. Where we start a new experiment. After that, we can select our experiment class type, which in our case would be Text Summarization. Once completed we can select the models to train the experiments on, the model selection provides us with various filters. By completing the selection of models we can upload our dataset to evaluate our experiment, we can upload the dataset as a zipped-up CSV file containing the text summarization articles. We need to provide the column name in which the articles are present. After running the experiment we can see the results tab show up, here you would also be able to see the latency of the models as well as the results of the experiment, for our use case we are able to see the table containing our input and the respective model summarized articles. We can also download the result in a zip file. Interesting results are can be evaluated for one such given article

  Apple's 2022 MacBook Air with an updated design, M2 processor, MagSafe charging, and a slightly bigger display will soon be available for preorder, starting at $1,199. Apple says it will open preorders online tomorrow, July 8th, at 8AM ET / 5AM PT, and orders will begin shipping Friday, July 15th...

We get the following text summaries for the respective models: Pegasus Xsum :

Apple's latest laptop is now available to pre-order.

T5 Base CNN :

Apple's 2022 MacBook Air will be available for preorder starting at $1,199 . The chip has an updated design, M2 processor, MagSafe charging, and a slightly bigger display . Apple says it will open preorders online tomorrow, July 8th, at 8AM ET / 5AM PT .

Bart Large Xsum :

If you're looking for a new laptop this year, here's your chance.

Bart Base CNN :

Apple's 2022 MacBook Air will be available for preorder starting at $1,199 . It will have an updated design, M2 processor, MagSafe charging, and a slightly bigger display . Preorders will begin shipping Friday, July 15th .

The Bart Large Xsum gives us clickbaity summaries. We can do additional investigation around this particular model. As seen from the experiment it is easy to create and evaluate different models for our use case. You can share the experiment with your coworkers and on social media. Also, you can make a copy of the given experiment to enhance or modify the particular experiment.

Wish to create one? Head on over to Tiyaro !