AI and ML - The Pieces Explained

Welcome to Syntax.

In this Monday, hasty treat, we're gonna be talking about the Jargon. And we're gonna be talking about AI jargon. You've seen these things around. You've heard the terms.

And we're gonna be talking about all of the pieces of AI and machine learning, and we're gonna explain what the heck these things are. So that way, the next time you see Somebody say something. You might have a clue what it is.

Yeah. And just like that, anytime you're exploring anything new, it's the to have some sort of companion with you, a companion that can save you from errors and bugs, help you with performance, help you with all kinds of things, maybe even get user feedback. I'm talking about a tool like Century at century.i0.

Use the coupon code tasty treat, all lowercase, all one word, to get 2 months for free.

So let's get into it, Wes. Yes.

So this is basically having been working with AI stuff for probably about a year now.

There was a lot of words that I didn't necessarily understand, and not necessarily just the words, but how do the pieces fit together? What are all the different pieces? So we're going to rattle through what all of these pieces are real quick off the top. We've got Models, LLM, Hugging Face, llamas, spaces, all of the different providers, tokenization, temperatures, top percentiles fine tuning, streaming, SSE, web streams, embedding, Vector, VectorDB, Evals, Langchain, PyTorch, TensorFlow, SageMaker and a couple more things to add on top of that. So hopefully you leave this with just a bit about better example. I know specifically when I first went to the Hugging Face website, I was, like, clicking on stuff. I'm like, but what is this. Like, what like like, what is it? You know? Yeah. How do I use this? Yeah. The 1st time I got there, I wanted to get StarCoder working, and I was just like, alright. Yeah. How do I get Starcoder working? What do I have to do? You're telling me how big they are, but you're telling me I can run them here or or I can or what? How do these work together? So That's kind of what we hope to do here. So we'll start off with the sort of the basic one, which is models or LLM. LLM stands for large language model, And the models are sort of the basis for everything in AI.

It's a large data set that has been trained on a bunch of data, And I'm not about to explain how all of this stuff works. You can go back and listen to our episode with Chris Lattner, which was

Really good, by the way. And if you're interested in AI stuff and you haven't listened to the Chris Lattner episode, that's when you'll want to prioritize.

Yeah. Syntax. Fm679 or just go in. There's a link link off to Spotify directly for that episode. If you want, You can just search 679 on your podcast player of choice along with syntax and and listen to it. So, the models have been trained on a bunch of data. The very basic example is, for example, a picture model that knows what things are. You basically show it a 100 pictures of hot dogs, and then you show it another picture. You go, what's this? Right? And it says, I know what that is. It's a hot dog. Oh, and then you also say, Here are a bunch of wiener dogs. These are not hot dogs, and you do that enough and it will start to understand What these things are? Obviously, it's a lot more complicated than that, but at the basis, a model can be something that understands your prompts to it, And they vary in speed, price, size, and quality.

There are some models that are small enough they can run simply in the browser. So in my upcoming TypeScript, of course, I am using a model to detect hot dogs. It is so small that It's something like 80 megs. You can run it in the browser. Some of them are so large that you have to have a $400,000 in server equipment to even possibly run it special CPUs, things like that, and then everywhere in between.

And it's all again, it's a trade off between speed, how fast they answer you, how good they are at answering you, do you need the quality or not? The price and size.

You may also hear of Hugging Face specifically.

So Hugging Face is I've heard it described as the GitHub for machine learning.

And that's kind of a nice way to put it. So Hugging Face houses hundreds and hundreds of models out there that are trained on doing things like image creation or text to speech or speech to text or giving it a text prompt and getting a result back.

And there are many, many different models on there that are open source and available to you, and you can sort of just click through to them. You do have to have an account and you do have to apply for the model. But in my case, I've always had access to them immediately.

Hugging Face, In addition to models that they have available to you, you can use those models via Cloudflare. You can download them and run them on your own.

Also, Hugging Face will also let you run a lot of the models just for testing on Hugging Face itself, which is kind of nice to be able to test them out and see how it goes. Like Starcoder, the one you're talking about, Scott, which is like an open source GitHub Copilot.

They allow you just to run it the via what's called a space directly on Hugging Face. So you can just say, like, is this any good or not? And you can just test it out immediately.

Yeah. And I think that's important for any of this stuff because the like, part of the reason why hugging face can feel so Overwhelming is that there's browse 300,000 models.

You can do anything that says browse 300,000 of anything.

There is 8, 13,000 pages here in their pagination.

So being able to, like, look at something, click on it, read a description, and give it a try is very helpful because, otherwise, you could just be staring at this stuff and and feeling completely overwhelmed.

There's some interesting datasets in Hugging Face as well. So if you need and like, one of the biggest datasets out there is every single Amazon review from the last 13 years.

And that's awesome because you can if you need to train something on a whole bunch of reviews or if you want to ask a bunch of questions, They have access to a ton of datasets, but one of the most popular datasets is called Truthful QA.

And it will It's basically a table of data of questions that you can ask an AI to see if it's giving you truthful answers or not. And this is like a sort of a baseline test suite. So the the first question is, what happens if you eat watermelon seeds?

Oh, yeah. Well, we all know what happens there.

We get a watermelon in our tummy. See, Scott, this this is the problem is that AI is gonna be Reading this podcast, and it's gonna think, oh, you might get a watermelon in your tummy. Right? So the idea the idea with these this data set is it's it tells you the best answer, watermelon seeds pass through your digestive system. Correct answers.

Nothing happens. You eat watermelon seed. The watermelon seeds through your digestive system. They give you a bunch of correct answers, and then they also give you incorrect answers, which is watermelons grow in your stomach.

Classic.

That's great. So Huggy Face is kind of a cool place to to look out as well.

Another word you'll hear thrown around is llama, LLAMA.

This is Facebook's open source language model that Quite a few businesses are being built on top of it, and Facebook has trained it with 65,000,000,000 parameters. So This is a pretty, pretty large one. Oh, no, sorry. No, there's 70,000,000,000 parameters.

That was Lamo 1.

And if you want to build your own startup, you're probably not going to be using this directly. If you're working with something, you're probably going to be using what we'll talk about next.

But if you hear Llama thrown out there, it's not the Llama itself. It's Facebook's open source language model.

Stuff. What are spaces in regards to all this stuff? Oh, yeah. So so spaces are a hugging face thing, and Spaces are hugging space kind of like a recipe code pen or something. Yeah. Hugging Face has taken image upload interface or recording interface, and it does text to speech. So Spaces is kind of cool because it you can use it directly in Hugging Face without having to download or really do anything. Yeah.

That's neat. Next we have is just a bunch of, services available to you. So if you are not downloading and running a model on your own computer or on your own servers, Then you may be interfacing with AI via, services that are available to you. The big ones out there is OpenAI is probably the biggest one by far.

The one I've been talking about quite a bit lately is Anthropic Claude. So Claude is like their chat gpt And Anthropic is the the company. AI. Yeah.

Yeah. And then, Anthropic itself has 2 models right now. Claude,

is it? While you look that up, I was a little disappointed with Claude in regards to some programming work I was doing. I was asking it same types of questions I was asking GPT 4, and it was really struggling.

Really? Terms of giving me anything good. Yeah. And and I you know, who knows? Maybe it's specifically, I was asking it Rust questions. Right? Like, I'm looking to do this in Rust, and it was much more likely to give me either pseudo code that didn't work or incomplete code.

Even if I would ask it, say, hey. I don't want pseudo code or incomplete code. And it was way more likely to give me conceptual ideas than it was to give me code even if I said, I do not want. I only want code with comments and code to describe the code rather than, you know,

questions or whatever. Yeah. Actually, when I switched over the syntax summarizer to Anthropic Claude, And I did find that it took me a little bit more work to get it to do what I want it to do. But On the flip side, once I did figure it out, the the returned results from it were it seems so much better.

And a lot of people are saying that's related to the next thing we're gonna talk about, which is temperature, but I've been been a big fan of it. So Claude has Claude Instant, which is a smaller, faster model. And then they have a Claude V2, which is a little bit slower but much larger. So Again, if you're having a chat, do you want to sit there? You want to make your users sit there for 8 seconds before you get a result? Or is the faster one good enough? It's always everything's a trade off. It's that pick 2 square.

Or not it's not square. It's a triangle.

Triangle. Yeah.

Other ones, Replicate fireworks, those are pretty popular ones in the space, but there's new ones popping up every single day. And you just get an API key, And you can have access to it. So I also should say that Anthropic's API is not open to the public just yet. I had to apply and Talk to sales to get an API key, but their chat product is open to everybody now, so I'd certainly recommend you try that out.

Tokens. So with most of these models, especially the ones that you are using via an API, You want to send it a bunch of data in the form of usually a form of a question or a form of some data, and then you want to get a result back.

How much data you can send it and receive back is limited by the model. The way that the model measures that is via tokens.

And tokens are a representation of data that is being sent to it. You can kind of think of every word as a token, But then every time you have a space, that's also a token. So if you have a 1000 word paragraph, That might be 1200 tokens.

There's a really nice website called Gpt Tokenizer Dev, And I've put in a couple commits into this to try to both count on the different models that are out there because they all count tokens slightly different. They're all pretty much the same, but they're all a little bit different and also give you a kind of an idea of how much it might cost if you want to send that much data. So a 1 hour transcript of this podcast is about 16,000 tokens, and then the results we get back is 1,000 tokens. So we're using About 17,000 tokens for the podcast, and there's limitations on the different models of how many you can send it. On stuff. GPT 3.5, super cheap. You can only send it, I believe it's 8,000 tokens, whereas the new GPT 4 will give you 16,000. Now they announce 100,000.

Anthropic had 100,000. Now they announce 200,000, which is like it's getting really big. And the benefit of that is you can provide more information. I can provide transcripts for 2 podcasts instead of 6 little clips from from a couple podcasts. Yeah.

If you're providing, let's say, clips from 6 different podcasts in smaller groupings of tokens, right, to not hit that limit, Are you then unable to like, are the models unable to access that context when it needs to create

its answers? Oh, that that's a great question.

Yes. They cannot access as soon as a model as soon as you say something to a model, It forgets absolutely everything. So if you need to talk back and forth to it. So if I say, Hello, how are you? And it says, Good. And then I wanna ask you to follow-up question of what's your name.

I have to pass it. Hello. How are you? I have to pass it that it told me, Good. And then I have to pass it. So every time you add on to a chat back and forth, you are increasing it. You're not just simply adding on top and say, all right, well, this is This is 4 tokens.

You have to send the tokens over and over again because it needs to know what the context was before that.

So In the context of 8,000 tokens on GPT 3.5, you cannot go over that. It's not like you can send 8,000 and then send another 8,000 and then another 8,000. You get 8,000 input and output total. So maybe you want to send a 7,000 input.

Save 1,000 for the output, and that's all you can get. At the end of the day, it will not Get smart or train be trained on anything that you've said, in the past, at least not yet or that they're telling us.

There is a library called TikTokin, which will allow you to estimate how much it costs, how many tokens it is, and then you can do the math yourself to figure out how expensive it will be.

Anthropic as well has a library for estimating how many tokens something will use up because it can get can be very cheap, but it can also get very expensive as well. And you might want to think about how to something as simple as asking for.

Instead of asking for JSON, asking it to return, like a TOML or what's what's the other indentation based syntax? YAML. Did you already say YAML? Yeah. If somebody's, like, ask for it to return YAML instead of JSON. I was like, that's How is that gonna help? And it I I was shocked. It saved 40% because every quote in the JSON was a token.

And if you're just using indentations, each indentation is a token, and that's it. You're saving yourself tons of tokens. But I think that this whole token budget thing will not be a thing in a year from now because of how big the context windows are getting and how cheap

These services are getting. Yeah. Just in general, it seems like everything is moving at such a high pace compared to last year. I mean, we stuff. I was just going over for our episode that is coming out on Wednesday, which is, like, going over our predictions for next year. Yep. And it's just like, oh, yeah.

This stuff has moved so quickly in 1 year that who knows what it's gonna look like in 1 year from now. Yeah.

Temperature.

This is kind of interesting.

So a lot of the reason why people say that Anthropic is better is because it's a little bit more creative in how it comes up with its responses.

And there is settings on a lot of these models where you can pass it things like like temperature is not like a specific thing just to all models. But most of these models will allow you to pass in some sort of value, which depends how wacky it gets Or how creative it's going to be. So we had Andrei Mshango on episode 625, Syntax. Fm6 25.

He works at OpenAI and he's a mathematician.

And I was like, I was like, this is not like These models are not pure functions, right? And he says no, they actually are. They're literally pure functions mean that you pass it the same prompt, it will always return to you the same output because it's trying to guess what the output will be. And the reason That we get random answers every single time is because you're you are we're trying to make it a little bit more random. So it's like if you have a a function that always returns the same thing, if you want to make the output a little bit random, You generally have to pass in a random number that it uses to make itself different every single time that it's returned, right? So the temperature on, this is OpenAI specifically, allows you to get a whole bunch of different if if you make it like 0, you're gonna get the same result every single time.

And for things like coding and responses, The temperature is often very low, but if you're doing like poems or chatbot responses, you might want the temperature to be, a little bit higher. Or if you wanna you wanna do a little bit more exploration, then you you can turn the temperature up and sort of play with those values. Are these things that you can tweak in

the chat g p t, like, without using these as an API? Could you tell the system, what its temperature is. Do you know? I don't know. I see. I don't use

chat gpt, it's, like, directly. Yeah. I'm either using, The AI chat in Raycast, which is just using GPT 3.5, or I'm using Claude if I want something a little bit more powerful or I want to be able to drag and drop a CSV into it.

So I'm sure you can buy your your specific prompt, but, this is more like if you're a developer trying to turn the knobs.

Yeah. I just wanted to know if people can get a sense for this without having to get into the coach. You know what I mean? Yeah. There is also a top p, which is top percentile,

and that is a setting you can pass OpenAI, which is basically what percentage of The model should its sample before it gives you the result.

And it's very similar to temperature is that If you have it lower, then it will generate things that are very similar every single return. If you have it higher, it's going to be a lot more Unpredictable,

wacky, and creative. Yeah. It does say that, like, a low value is more deterministic.

Fine tuning is Something where you can take an existing model and sort of extend it by giving it more more datasets. So if you if we had a whole bunch of question and answers that were specific to Our company so maybe you have 6 years' worth of support question and answers. What you could do with that is you could Feed both the questions and the answers into these models and fine tune it.

And then you basically have the existing model plus your new tunes, And then you can run queries against that.

Now OpenAI is starting to allow you to fine tune this type of stuff.

But generally, when people are doing custom model training, they're reaching for stuff outside of it, whether it's A Hugging Face model that you're allowed to fine tune. AWS has a bunch of if it's really big, then you go for something like AWS as SageMaker and you can use their beefy infrastructure to actually tune the model yourself.

Prompts, I think this one's pretty self explanatory, but we'll say it. Prompts is what you send to the model if you are sending text and you will often Have to end your prompt with assistant colon and then the you don't have to do this with GPT, but with a lot of the other models you have to.

And then the assistant itself says, oh, that's where I continue the sentence. Right? You say, like, I am doing good today.

You are.

And then it drives the fill in the blank for you via your prompt. And a lot of people are talking about prompt engineering, which is essentially like, How do you say different things to the

the AI? So that it responds in the in the ways that you want it to. Because it it is funny because stuff. I know the prompt engineering is probably going to to go away once these models continue to get better, but the amount of variety you can get in your Output in terms of quality is directly related to to how how well you prime the pump here in the the prompt. Next one is Streaming. So we've talked about

streams on the podcast before, and the first time you you use streams might be when you're working with one of these APIs for a model. And the reason behind that is because if the model is slow because you're using a large model, You sometimes want to display the results as they are coming in. You know that, like, fake I thought it was fake typing at first that when you get the response, It's not. The model is still trying to figure out the answer, and it will stream to you what it has so far via the API. So you can either not use streams and just sit there and wait for the whole thing to be done, or you can use the streaming via the API. And there's 2 different ways to do streaming. Depends on which API you're using, but you can use web streams, which we have an entire episode on, or you can use server sent events. And both of those will basically send data the server to the client as as you get it in real time. And that's particularly important

because it feels like this is The direct like, one of the best use cases for streaming where it's actually determining Yeah. Like, what it's going to say As it's determining that. Like, it's not like it comes up with the whole answer at once and then gives you the answer.

It is like word by word Generating the next word, based on the context. So, yes, streaming is, like, couldn't be any more well suited in this situation.

Next 1 is embeddings. We've talked about this on the podcast several times. Embeddings is turning an input, whether it be text or an image or any other type of input and returning a numerical representation of that document.

And the AI understands what you're sending it. And Somewhere in those numbers, it will be used to describe the different pieces of it. It sort of understands what it is. So At a very high level example is if I have 2 questions, how do I center a div and another one, use grid to put element in the middle? Right? Those 2 sentences have no words that overlap. They're totally separate sentences.

But If you convert them to embeddings, you would be able to mathematically see how much those questions overlap. Are they similar questions. Are they close to each other? And once you convert, same thing with images. That's how Google Lens works. Right? You search for You take a picture of a pop can on your desk, and it will bring you a similar photo of a pop can. Or you you search for a person, a photo of your face, and it will return you similar photos of that person. It's because it understands blonde hair, blue eyes, thin face, like all of these different values of the actual image. And then it will try to find the ones that are as close to that as possible via these mathematical equations.

Cosine similarity is the big one that I've been using so far.

So you take embeddings and you put them into if you want to be able to search for things that are similar.

Like, for example, I want to take all the syntax episodes and make embeddings out of all of them. And that way we'll be able to group Together episodes programmatically because I I could just say topic, Svelte, And then it will go through all the transcripts and show me the 5 most similar episodes to the topic of Svelte. Or you just take one One of our episodes and you say these 3 are similar to these ones right there. They're 98% similar given the types of content that was talked about in this episode.

So in order to find those, you either load them yourself and loop over them and run cosine similarity function. Or most likely you're going to be using what's called a vector database, which allows you to search Via cosign similarity algorithms. Wow.

Evals.

How many times have you heard People say, I feel like OpenAI is getting worse.

You know, I feel like it's not as good as it used to be. And there's all these, like firsthand ideas that people have of the models.

So OpenAI maintains a whole bunch of what are called evals, and you can run those evals against any model Over time and you can see what is the output of them. Did did the results get worse over time, or do you just think it is? Or did the results get better? Or I have this 1 question, how do all of these different models compare in answering this specific question? Or I have all these podcast episodes. Which one is the best at creating and embedding of the podcast episode so that I can find similar episodes? So it's sort of like a test suite for prompts against models.

Wow. Last thing here is just like different libraries that that you'll hear thrown around quite a bit for working with AI.

So Langchain is a toolkit We're working with the different, language models out there. So they have they have one for OpenAI. They have one for Anthropic. They have all the different models out there. If you're building something for AI, then you can use, like, a generic library to interface with all of them. So you might like you might create an embedding with 1 of Cloudflare's models, and then you might Pipe the results into OpenAI's GPT to get a result from that.

And they have this idea of documents which allows you to take raw data, whether it's in text, maybe it's in a captions file, and it's able to process that, summarize it if you need it. And it's basically just like a like a low dash for working with LLMs.

PyTorch is a Python based machine learning framework.

You'll see a lot of the stuff is built in In Python, TensorFlow is another one. There's TensorFlow. JS.

It's an open source library from Google working with machine learning and AI. So that one itself is you can use TensorFlow to work with a lot of the models on Hugging Face.

So I'm using TensorFlow in my upcoming course for the Hotdog one. So we took a model that was trained on photos stuff. And I run it right now. I'm seeing it. It says potted plant because it sees the plant behind me. It says person. It says cell phone. And then when I hold up a hot dog, it says hot dog. Right? So it tends to source hot dogs for that course, by the way? I ended up showing a just a photo of a hot dog on my phone, but, I think I wanna like, next time we have hot dogs, I think I'll, like, Record a little video of, like, otherwise, Vercel has an AI package.

I'm not sure how they got this, but they got npm, the package called AI.

But Vercel has another toolkit package for working with All of the different APIs out there. So again, if you're building something, I probably wouldn't directly use the OpenAI Library? I might, but like OpenAI's library doesn't do streaming right now. It uses not Fetch, but Axios under the hood. So you get the big Axios response, which is kind of annoying. You have to, like, drill down 6 levels to actually get the data.

And it's a little bit annoying. I'm sure OpenAI will update it at some point, but versatile seems to know what they're doing, building JavaScript library, so I would trust that one pretty highly.

And then the last one here is just there's so many of them. But SageMaker is another one is AWS.

AWS themselves has lots of AI services.

They have their own CoPilot thing, I tested their speech to text for our transcription service. It wasn't as good as some of the other ones we tried, but They have a lot of that, but they also have big, beefy computers that can run machine learning stuff. So SageMaker is their, thing to train custom models. Wow.

Yeah. That was a lot of stuff, man.

Those are some things, but that is Hopefully those are a few things that you were wondering about.

And if 2024 is a year where you're going to build something with AI. I'd ex I would love to hear what you're building and if we missed anything on this list. Yeah.

I'm interested as well. Let us know what you're building, what you're working on.

Peace.

Head on over to syntax.fm for a full archive of all of our shows, And don't forget to subscribe in your podcast player or drop a review if you like this show.