Table of Content:
- Introduction
- Useful Links
- When to use Einstein Language vs Custom AI Tool
- The Dataset
- Handling Out of Vocabulary Words and Phrases
- Creating a Model
- Step 1 – Create your data file
- Step 2 – Upload the Data Set to Einstein
- Step 3 – Train the Model
- Step 4 – Test
- Building the Lightning Component
- Test Results – Comparing Sentiment, Intent and Sagemaker’s Blazing Text
- Final Thoughts
Introduction
In this module, I will talk through creation of Machine Learning models using the Einstein Intent and Sentiment APIs and exposing them within Salesforce.
Intent and Sentiment are services available under Einstein Language – the NLP module of Salesforce Einstein.
They both allow analysis of text. Intent is a bit more generic allowing you flexibility of categories(Intent) text can fit into, whereas Sentiment is specifically tailored to classify text into Positive, Negative and Neutral Sentiments
The organization I was working on was in Lightning so I wanted to create a single reusable Lightning Component which I could drop into multiple screens – Accounts, Cases, Tasks – and have it work without issue
I will also talk through considerations around datasets, some of my experiences with datasets and test results from the specific dataset I used to train my models.
Finally, in my experience, choosing an AI tool does not happen in isolation and the question I get asked a lot is ‘How does Einstein compare against Product XYZ’. In that spirit, I also trained a model using the same dataset in AWS Sagemaker – another commonly used AI tool – using Sagemaker’s Blazing Text Algorithm – a standard NLP algorithm. I will provide comparisons on how Einstein Language compares to that tool throughout the article.
My code to train the model in Sagemaker is here: https://github.com/vvr-rao/BlazingText-for-Text-Classification
For an example on how to deploy an model in Sagemaker and call it from Salesforce, I have a blog post here: https://vvrrao.home.blog/
I also trained another NLP model (though with a different data set) and was able to deploy it on AWS EKS: https://github.com/vvr-rao/NLP-on-AWS-EKS. While it takes more effort, it was well worth the effort as I could add additional functionality e.g. leverage open source libraries like Spacy to do things like Named Entity Extraction
Useful Links
- https://trailhead.salesforce.com/en/modules/einstein_intent_basics – Salesforce Trailhead – tutorial on Einstein Intent. This provides the basic grounding on Einstein Language and how to create, train and use a model.
- https://github.com/salesforceidentity/jwt – Salesforce provided JWT libraries. You will need these in the code
My code is also available on github at : https://github.com/vvr-rao/SFDC-Einstein-Intent-Lightning-Comp
Back To TopWhen to use Einstein Language vs Custom AI Tool
Einstein Language is obviously a good fit for Salesforce. The question is when to use it vs using a Custom AI tool.
It really boils down to a cost benefit analysis. Einstein is easy to implement but you don’t have the control in configuration to fine tune your model.
Blazing Text gave me greater control to tune the parameters (epochs, learning rate. Test/validation data split etc.) and there are options to publish the model over Lambda.
With Einstein, if your model is not providing an adequate accuracy, you are pretty much stuck with finding newer or better data. You might get great results, or it might not be as good as you hoped. As such, your data set is crucial. Let’s start there…
Back To TopThe Dataset
For a pet project, there are plenty of datasets available publicly for amateur model builders. In practice, you are better off getting a data set based on your own business needs.
In my example I used the data set from this site: http://help.sentiment140.com/for-students
This is a set of 1.6 million tweets classified as Positive or Negative (the description on the site is misleading – it states Positive, Negative and Neutral but Neutral tweets are not present)
Of course, the code I provided can be used with other data sets – the results will be based on this data set. Your results may vary based on the data set you use.
If you are looking for datasets, Kaggle (link to the right) is a good place to look and here is another good site to get you started:
https://blog.cambridgespark.com/50-free-machine-learning-datasets-sentiment-analysis-b9388f79c124
I trained my initial models using datasets from these sites and got fairly accurate results from generic queries.
Back To TopHandling Out of Vocabulary Words and Phrases
Any NLP tool is vulnerable to Out-Of-Vocabulary Problems and as we have no way of modifying the model with Einstein (more on that later), we have no way to control how OOV words are handled.
In general, Einstein seems to be able to pick up what an OOV word means based on context but it is inconsistent. For Example…
This…
A person called asking for Information.
…..gets a somewhat Positive probability rating from the generic models.(between 70% to 90%) positive
However, start adding industry specific information, e.g.
A HCP called asking for Information
…. and the model gets a bit more confused because a “HCP” probably does not exist too much in the generic dataset. A more industry specific data set for Life Sciences or Healthcare would probably be more accurate.
Taking it further, I tried…
A HCP called asking for Information on our Company Product X
I was expecting this to throw the model for a loop. That is because ‘Product X’ is specific to a company and the only dataset with that kind of data is something you would generate yourself. Surprisingly, Einstein was able to figure out what Product X meant based on the context.
It doesn’t always work as expected though:
One does not simply walk into Mordor
Gives a somewhat negative Sentiment with Einstein (and an ambiguous result with Blazing Text).
However,
One does not simply walk into Chicago
Gives a positive Sentiment.
In short – Einstein seems to have some capability to infer meaning of OOV words based on context. This was something I did not do when I trained my model in Blazing Text.
I then tried an OOV Phrase:
A HCP called and told us to go fly a kite
Perhaps understandably, Einstein classified the above as generally positive.
Control Over the Model
As mentioned earlier. the answer here is simple, with Einstein, we have no control over the model, unlike a custom machine learning algorithm. If you train the model and find your results are not as good as you wish, the only recourse is to find new or better data. You do not have the option to switch algorithms or tune hyper-parameters.
However, Einstein does provide some tailored models.
Einstein Sentiment seems to be a tailored version of Einstein Intent optimized for it’s use. I found that the Sentiment Model was more accurate than the Intent Model for predicting Sentiment (as expected) and performed better than my own attempt using Blazing Text. I have some examples later in this article. However, we don’t have much documentation on the inner workings of the Einstein models.
Getting an environment.
Let’s start building out a POC. We start by getting an environment.
You need an environment with Einstein Language installed. You can sign up for one here: https://api.einstein.ai/signup
When you sign up, you will get a .PEM file. KEEP IT SAFE!!
You can use the .PEM file to get a Authorization Token to access the API. Later in this earticle, we will see how to get the token programatically.
For now, you can go to he following link and generate one: https://api.einstein.ai/token
Back To TopCreating a Model
Training the model in Einstein involves 3 steps – upload the data set to Einstein, Train the model, Test. The steps are the same whether you use Einstein Intent or Sentiment though the endpoints are different. I have given both examples below. (I ran the data set through both models – this is technically feasible though the results with Sentiment are more accurate)
However, there is more to it than that – typically you will need to clean up and massage the data. Einstein expects the data to be in .CSV with a “text”, “label” (no headers). You will need to clean up your data first to meet Einstein’s requirements. That will be step 1…
Step 1 – Create your data file
The data set in this example, uses Twitter data so I felt it was appropriate to clean up things like twitter handles. Also, Einstein Sentiment needs the labels to be specific values so the code sets that up as well.
I used Python, leveraging the standard libraries like Numpy and Pandas – here is my code(note, this is specific to this dataset). The full code I used is on my github at the following link: https://github.com/vvr-rao/SFDC-Einstein-Language-Lightning-Comp
#===Libraries to Import===
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.utils import shuffle
import re
import csv
import nltk
nltk.download('punkt')
#===Set up Data frame===
#Note: Raw_data.csv is the source data file from http://help.sentiment140.com/for-students
df = pd.read_csv('Raw_data.csv', delimiter=',',encoding="latin1", header=None, usecols=[0,5])
df = shuffle(df)
#===Couple of Helper functions===
#1) convert 0, 2 & 4 to text labels (Einstein Sentiment needs these)
# 0 = negative, 2 = neutral, 4 = positive. Issue with data set - only 0 and 4 are available
index_to_label = {}
index_to_label[0] = 'Negative'
index_to_label[2] = 'Neutral'
index_to_label[4] = 'Positive'
print(index_to_label)
print(index_to_label[4])
# set Labels appropriately
def labelize(inp):
return (index_to_label[inp])
#2) Clean up the tweet body. There is more complex logic which can be built but I wanted to take out special characters and twitter handles
def tokenize(inpstr):
Tweetstr = inpstr
Tweetstr = re.sub('@[^\s]+','',Tweetstr)
return(Tweetstr.lower())
#===Apply cleansing logic ===
df[df.columns[0]] = df[df.columns[0]].apply(tokenize) #standardizes first column to remove characters and words I don't want
df[df.columns[1]] = df[df.columns[1]].apply(labelize) #standardizes labels in second column
#===Generate My file ===
Note: I reserved 5% of my data for testing
rows = df.shape[0]
train = int(.95 * rows)
test = rows-train
df.iloc[:train].to_csv('training_EinsteinSentiment.csv', sep = ',', header=False, index=False)
df.iloc[train:].to_csv('test_EinsteinSentiment.csv', sep =',', header=False, index=False)
This should get you the data in the correct format
Step 2 – Upload the Data Set to Einstein
You create a model using CURL. Create it using one of the following commands depending on whether you are planning to create a Sentiment or an Intent Model. (the difference is the type =text-intent vs text-sentiment )
Intent
curl -X POST -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "type=text-intent" -F "path=<location_of_dataset>" https://api.einstein.ai/v2/language/datasets/uploadSentiment
curl -X POST -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "type=type=text-sentiment" -F "path=<location_of_dataset>" https://api.einstein.ai/v2/language/datasets/uploadAs I mentioned earlier, you could technically train the Intent model to return Positive-Negative-Neutral values but, in practice, for the same data set, you get better results using the Sentiment model for Sentiment (Positive-Negative-Neutral)
The path to your file needs to be publicly accessible over http. Dropping the file on your machine will not work. I placed my file on AWS S3 and made it public temporarily.
EDIT: I did speak to somebody who mentioned that you can in fact read the file from your local machine but need a specific format in the path. I did not get the format though.
The output of the above will be a JSON. The most crucial element will be the id – which is your Dataset id and used in subsequent calls.
Step 3 – Train the Model
You train the model using the following command.
curl -X POST -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" -H "Content-Type: multipart/form-data" -F "name=forload.csv" -F "datasetId=<dataset_id>" https://api.einstein.ai/v2/language/trainYou plug in the dataset id above. The output contains a lot of things one of which is the Model Id which you will use to use the model
If you take a look at the file, there is some other information available. Here is a sample. Note the parameters like epochs, learning rate etc. You cannot set any of these, Einstein choosing what is appropriate for you:
{
"datasetId": 1148414,
"datasetVersionId": 213507,
"name": "test_EinsteinSentiment.csv",
"status": "SUCCEEDED",
"progress": 1,
"createdAt": "2019-11-11T16:30:20.000+0000",
"updatedAt": "2019-11-11T17:00:06.000+0000",
"modelId": "XXXXXXXX",
"learningRate": 0,
"epochs": 1000,
"modelType": "text-sentiment",
"language": "en_US",
"object": "training",
"trainParams": null,
"trainStats": {
"labels": 2,
"examples": 77986,
"averageF1": 0.8008214845748466,
"totalTime": "00:29:39:921",
"transforms": null,
"trainingTime": "00:27:45:468",
"earlyStopping": true,
"lastEpochDone": 45,
"modelSaveTime": "00:00:00:375",
"testSplitSize": 15728,
"trainSplitSize": 62258,
"datasetLoadTime": "00:01:54:452",
"preProcessStats": null,
"postProcessStats": null
}
}
Training can take a while depending on dataset size(a dataset with 80k records took about 30 minutes for me though I imagine results vary based on server load). You can check the status using the below:
curl -X GET -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" https://api.einstein.ai/v2/language/train/<MODEL_ID>Step 4 – Test
You can start using your model once the above command returns a status of SUCCEEDED
You do this by sending the model id and data you wish to analyze via HTTP POST to the API endpoint. The endpoints for Intent and Sentiment are:
Intent: https://api.einstein.ai/v2/language/intent
Sentiment: https://api.einstein.ai/v2/language/sentiment
Here are the CURL commands:
curl -X POST -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" -H "Content-type: application/json" -d @intentjson.txt https://api.einstein.ai/v2/language/intent
curl -X POST -H "Authorization: Bearer <TOKEN>" -H "Cache-Control: no-cache" -H "Content-type: application/json" -d @sentimentjson.txt https://api.einstein.ai/v2/language/sentimentThe input json will be of the format
{ "modelId": "MODELID", "document": "I would like to buy shoes(text you want to analyze)" }Remember the Model Id needs to be of type ‘text-intent’ or ‘text-sentiment’ respectively
If things go well, you should expect a response similar to the below;
{
"probabilities":[{"label":"Positive","probability":0.9065981},{"label":"Negative","probability":0.060226},{"label":"Neutral","probability":0.033175863}],
"object":"predictresponse"
}
Building the Lightning Component
With the model good to go, we need to expose this in some way. I decided to use a Lightning Component and make it generic enough to drop the Lightning component into any record page.
My goal was to give the users a button which, on click, would send a pre-identified free text field on each of my target objects to the Language API and return the results of the analysis. At a high level I was going to do this:
Step 1 – Generating the Token.
You can generate a token using the .PEM file you generated when you created the Einstein account. (told you to keep it safe).
If you lost it you can get a new one using the ‘Lost My Private Key’ link.
You will need to import the JWT and JWTBearerFlow classes from here: https://github.com/salesforceidentity/jwt
In the example below, I stored my .PEM in a Salesforce File
public static String getAccessToken() {
ContentVersion myPem64 = [select VersionData FROM ContentVersion WHERE title = 'einstein_platform' AND IsLatest = true];
String myKey = myPem64.VersionData.tostring();
myKey = myKey.replace( '-----BEGIN RSA PRIVATE KEY-----', '' );
myKey = myKey.replace( '-----END RSA PRIVATE KEY-----', '' );
myKey = myKey.replace( '\n', '' );
JWT jwt = new JWT( 'RS256' );
jwt.pkcs8 = myKey;
jwt.iss = 'developer.force.com';
jwt.sub = '<EMAIL>';
jwt.aud = 'https://api.einstein.ai/v2/oauth2/token';
jwt.exp = String.valueOf( '30' );
String access_token = JWTBearerFlow.getAccessToken( 'https://api.einstein.ai/v2/oauth2/token', jwt );
return access_token;
}//getAccessToken
Step 2 – Call the Language API
You call the appropriate API endpoint by a simple HTTP POST request.
//make the call
string ModelId = '<MODEL_ID>';
string sResult;
Http http = new Http();
HttpRequest req = new HttpRequest();
req.setMethod( 'POST' );
req.setEndpoint('https://api.einstein.ai/v2/language/intent'); //If you want to call the Sentiment API( 'https://api.einstein.ai/v2/language/sentiment');
req.setHeader( 'Authorization', 'Bearer ' + sToken);
req.setHeader( 'Content-type', 'application/json' );
String body = '{\"modelId\":\"'+ ModelId + '\",\"document\":\"' + sSearchText + '\"}';
req.setBody( body );
HTTPResponse httpRes = http.send(req);
system.debug (httpRes.getbody());
sResult = httpRes.getbody();
Step 3 – Parse the Response
You parse the response using the JSONParser. It is worth noting, I created a class to hold the Intent/Sentiment, Probability values. I planned to use <aura:iteration> to display the returns in my Component.
public class IntentWrapper {
@AuraEnabled public String Token {get;set;}
@AuraEnabled public String Prob {get;set;}
public IntentWrapper (string a, string b){
Token = a;
Prob = b;
}
}Here’s the Parsing logic:
JSONParser parser = JSON.createParser(httpRes.getBody());
integer i = 0;
string tempintent = '';
string tempprob = '';
while (parser.nextToken() != null){
system.debug('TOKEN: ' + parser.getCurrentToken());
system.debug(parser.getText());
if (parser.getCurrentToken() == JSONToken.VALUE_STRING) {
tempintent = parser.getText();
}
if (parser.getCurrentToken() == JSONToken.VALUE_NUMBER_FLOAT) {
tempprob = parser.getText();
i++;
retWrp.add(new IntentWrapper(tempintent,tempprob));
}
}STEP 4 – The Lightning Component
I need to embed the Lightning Component in Lightning Record Pages and needed to capture the record Ids. Therefore, my implements needed these 2: flexipage:availableForAllPageTypes,force:hasRecordId
As mentioned earlier, I planned to use a List of IntentWrapper to iterate over. which can be done like so:
<aura:attribute name="Intent" type="EinstenIntentWrapper.IntentWrapper[]" />
<table class="slds-table slds-table--bordered slds-table--striped slds-table--cell-buffer">
<thead>
<tr class="slds-text-heading--label">
<th scope="col" class="head">Intent</th>
<th scope="col" class="head">Probability</th>
</tr>
</thead>
<tbody>
<aura:iteration items="{!v.Intent}" var="item">
<tr>
<td>{!item.Token}</td>
<td>{!item.Prob}</td>
</tr>
</aura:iteration>
</tbody>
</table>
Step 5 – Putting it all together
Here is everything once I put it together.
Component:
<aura:component controller="EinstenIntentWrapper" implements="flexipage:availableForAllPageTypes,force:hasRecordId" access="global">
<aura:handler event="force:refreshView" action="{!c.isRefreshed}" />
<aura:attribute name="Intent" type="EinstenIntentWrapper.IntentWrapper[]" />
<div class="page">
<lightning:button variant="brand" label="Get Sentiment" onclick="{!c.getIntent1}" />
<h1>Result:</h1>
<div class="slds">
<table class="slds-table slds-table--bordered slds-table--striped slds-table--cell-buffer">
<thead>
<tr class="slds-text-heading--label">
<th scope="col" class="head">Intent</th>
<th scope="col" class="head">Probability</th>
</tr>
</thead>
<tbody>
<aura:iteration items="{!v.Intent}" var="item">
<tr>
<td>{!item.Token}</td>
<td>{!item.Prob}</td>
</tr>
</aura:iteration>
</tbody>
</table>
</div>
</div>
</aura:component>
Client Side Controller:
({
getIntent1 : function(component, event, helper) {
var rid = component.get("v.recordId");
var action = component.get("c.getIntent");
action.setParams({RecId : rid});
var self = this;
action.setCallback(this, function(actionResult) {
var state = actionResult.getState();
if (component.isValid() && state === "SUCCESS") {
component.set("v.Intent", actionResult.getReturnValue());
}
else if (state === "ERROR") {
var errors = response.getError();
if (errors) {
if (errors[0] && errors[0].message) {
console.log("Error message: " + errors[0].message);
}
}
else {
console.log("Unknown Error");
}
}
});
$A.enqueueAction(action);
},
isRefreshed: function(component, event, helper) {
location.reload();
}
})
Server Side Controller:
public with sharing class EinstenIntentWrapper {
@AuraEnabled
public static List<IntentWrapper> getIntent(String RecId) {
list<IntentWrapper> retWrp = new list<IntentWrapper>();
//retrieve Notes
string sSearchText = '';
if (RecId.substring(0,3) == '001') {
account acc = [select FIELD_REDACTED__c from Account where id =:RecId];
sSearchText = acc.FIELD_REDACTED__c ;
}
if (RecId.substring(0,3) == '500') {
Case c = [select FIELD_REDACTED__c from Case where id =:RecId];
sSearchText = c.FIELD_REDACTED__c ;
}
if (RecId.substring(0,3) == '00T') {
Task t = [select Description from Task where id =:RecId];
sSearchText = t.Description;
}
//get the token
String sToken = getAccessToken();
//make the call
string sentimentModelId = '<MODEL_ID>'; //'CommunitySentiment';
string sResult;
Http http = new Http();
HttpRequest req = new HttpRequest();
req.setMethod( 'POST' );
req.setEndpoint('https://api.einstein.ai/v2/language/intent'); //( 'https://api.einstein.ai/v2/language/sentiment');
req.setHeader( 'Authorization', 'Bearer ' + sToken);
req.setHeader( 'Content-type', 'application/json' );
String body = '{\"modelId\":\"'+ sentimentModelId + '\",\"document\":\"' + sSearchText + '\"}';
req.setBody( body );
HTTPResponse httpRes = http.send(req);
system.debug (httpRes.getbody());
sResult = httpRes.getbody();
// parse the result
JSONParser parser = JSON.createParser(httpRes.getBody());
integer i = 0;
string tempintent = '';
string tempprob = '';
while (parser.nextToken() != null){
system.debug('TOKEN: ' + parser.getCurrentToken());
system.debug(parser.getText());
if (parser.getCurrentToken() == JSONToken.VALUE_STRING) {
tempintent = parser.getText();
}
if (parser.getCurrentToken() == JSONToken.VALUE_NUMBER_FLOAT) {
tempprob = parser.getText();
i++;
retWrp.add(new IntentWrapper(tempintent,tempprob));
}
}
return retWrp;
}
public static String getAccessToken() {
ContentVersion myPem64 = [select VersionData FROM ContentVersion WHERE title = 'einstein_platform' AND IsLatest = true];
String myKey = myPem64.VersionData.tostring();
myKey = myKey.replace( '-----BEGIN RSA PRIVATE KEY-----', '' );
myKey = myKey.replace( '-----END RSA PRIVATE KEY-----', '' );
myKey = myKey.replace( '\n', '' );
JWT jwt = new JWT( 'RS256' );
jwt.pkcs8 = myKey;
jwt.iss = 'developer.force.com';
jwt.sub = '<EMAIL>';
jwt.aud = 'https://api.einstein.ai/v2/oauth2/token';
jwt.exp = String.valueOf( '30' );
String access_token = JWTBearerFlow.getAccessToken( 'https://api.einstein.ai/v2/oauth2/token', jwt );
return access_token;
}//getAccessToken
public class IntentWrapper {
@AuraEnabled public String Token {get;set;}
@AuraEnabled public String Prob {get;set;}
public IntentWrapper (string a, string b){
Token = a;
Prob = b;
}
}
}FINAL
As you can see, I use the first 3 characters of the RecordId to identify the object and retrieve the appropriate text field. This component can be placed on the Account, Case and Task objects. It can be extended for other objects in a similar manner.
Test Results – Comparing Sentiment, Intent and Sagemaker’s Blazing Text
As promised. Here are the results of the testing. As I mentioned, I used the same data set and trained a model using the AWS Sagemaker Blazing Text algorithm. As a background, Blazing Text, in short, is an NLP algorithm provided out-of-the-box as part of AWS Sagemaker. It is based on Facebook’s fastText. In layman’s terms, it tries to understand language in a vocabulary using its proximity to other words.
FYI – if you are interesting to see how I tuned the model – I am working on a larger blog post on it – but, for now, I did post my code on my Github: https://github.com/vvr-rao/BlazingText-for-Text-Classification.
One note, you have a lot more control over the parameters in Sagemaker. I played around with a lot of parameters to get the best I could. Eventually the parameters I used were
Early Stopping : TRUE (Patience: 4)
Min Epochs: 5
Max Epochs: 45
Word nGrams: 3
Vector Dimensions: 100
Learning Rate: 0.001
Let’s see how it stacks up against Einstein Sentiment and Intent.
Here were the key differences I saw:
1) EPOCHS:
Einstein Sentiment and Intent ran for 45 epochs even with Early stopping set as TRUE.
For Blazing text, the maximum epochs run was 36 (I used early stopping as true with a Patience of 4 – so essentially the model ran stopped improving after 5 epochs)
One note, I chose a learning rate of 0.001 in my model which might have driven this. Based on the response file I got from the Einstein Training job, it looks like it uses a Learning Rate of ‘0’ which does not really make sense to me. Sagemaker, of course gives much more control over parameters. If you look at my githib, you will see the validation accuracy of my model in .8179 with the above parameters. I plan to play around with it more to get better results
2) TRAINING DATA
With Sagemaker – I used 70% percent of my data for Training and 30% for validation – you have to choose this manually in Sagemaker.
Einstein automatically used around 20% (the number varies a bit) of the data for Validation.
3) TIME TO RUN
Einstein took a lot longer to run (about 30 minutes for 80,000 training records) My training job with 1.6 million records never completed. I keep getting a HTTP 504 error – downloading dataset for training :Invalid response [504] – which is odd considering that the download is happening from the Salesforce internal servers. I’m putting that down to me running these jobs using trial licenses since I have definitely been told that large files of 1.6 million records can definitely be supported. However, even with this dataset – I was getting more accurate results that Blazing Text.
Blazing Text was done in <10 minutes. Note, however, that there were far fewer epochs with Blazing Text (this would be dependent on the instance types you choose. However, I choose the cheapest I could find)
4) ACCURACY:
The key thing is the accuracy of predictions. Here are the results of some phrases run through each model. As you can see, there does not seem to be a clear winner between Sagemaker and Einstein Sentiment (the numbers returned are different but – as with any model – the question of whether one set up numbers is better is debatable):
A few Notes….
Unsurprisingly, Sentiment was a lot more accurate than Intent at identifying Sentiment than Intent.
Neither model seemed to understand the phrase “go fly a kite” which can boil down to having too many OOV words or neither model being good at handling sarcasm
Notice how Einstein handled the OOV word – HCP? Interestingly, while Einstein was getting confused with HCP, it seems to understand that ProductX and QETUOD meant something even though they were also OOV words.
What does that mean?
As I mentioned earlier, Einstein Sentiment vs Sagemaker Blazing Text is a bit up for debate
With Sagemaker, you have access to a multitude of algorithms. You also have visibility to the algorithms and parameters used. With Einstein, I still am not sure what the underlying logic is. You can also input much more data with Sagemaker (my Einstein job handled about 80k records whereas Sagemaker breezed through all 1.6 million records without a problem
It would be interesting to try other NLP use cases (i.e. not Sentiment) and see how Blazing Text fares vs Einstein Intent.
Back To TopFinal Thoughts:
The Einstein portion of the module was surprising easy to implement. That part took me just a couple of evenings to build out(Sagemaker took several weeks and training my own model in Python took me months of study to get something usable). I guess that highlights the key selling point of Einstein – it is something which is quick and easy to implement as long as you have the data for it.he other great thing is that it is fairly easy to get onto Trailhead and spin up a Playground Org to see how Einstein works with your specific dataset.
In summary, while Einstein is not the best AI product out there – it fills a solid niche in the market. It does provide a fairly good results as well and is definitely worth considering, especially if you already use Salesforce.
I mentioned a few times, I am also working on a POC with AWS Sagemaker and am looking forward to see how its algorithms compare to other Einstein features..
Till then, if you were following along and building out your own model, I would love to hear your experiences. As always, comments and feedback are welcome
Back To Top
Venkat's Playspace 