AWS AI & Machine Learning Podcast

Episode 11: XGBoost special

February 24, 2020 Julien Simon Season 1 Episode 11
AWS AI & Machine Learning Podcast
Episode 11: XGBoost special
Chapters
AWS AI & Machine Learning Podcast
Episode 11: XGBoost special
Feb 24, 2020 Season 1 Episode 11
Julien Simon

In this episode, I talk about XGBoost 1.0, a major milestone for this very popular algorithm. Then, I discuss the three options you have for running XGBoost on Amazon SageMaker: built-in algo, built-in framework, and bring your own container. Code included, of course!

⭐️⭐️⭐️ Don't forget to subscribe to be notified of future episodes ⭐️⭐️⭐️

Additional resources mentioned in the podcast:
* XGBoost built-in algo: https://gitlab.com/juliensimon/ent321
* XGBoost built-in framework: https://gitlab.com/juliensimon/dlnotebooks/-/blob/master/sagemaker/09-XGBoost-script-mode.ipynb
* BYO with Scikit-learn: https://github.com/awslabs/amazon-sagemaker-examples/blob/master/advanced_functionality/scikit_bring_your_own/scikit_bring_your_own.ipynb
* Deploying XGBoost with mlflow: https://youtu.be/jpZSp9O8_ew
* New model format: https://xgboost.readthedocs.io/en/latest/tutorials/saving_model.html
* Converting pickled models: https://github.com/dmlc/xgboost/blob/master/doc/python/convert_090to100.py

This podcast is also available in video at https://youtu.be/w0F4z0dMdzI.

For more content, follow me on:
* Medium https://medium.com/@julsimon
* Twitter https://twitter.com/@julsimon

Show Notes Transcript

In this episode, I talk about XGBoost 1.0, a major milestone for this very popular algorithm. Then, I discuss the three options you have for running XGBoost on Amazon SageMaker: built-in algo, built-in framework, and bring your own container. Code included, of course!

⭐️⭐️⭐️ Don't forget to subscribe to be notified of future episodes ⭐️⭐️⭐️

Additional resources mentioned in the podcast:
* XGBoost built-in algo: https://gitlab.com/juliensimon/ent321
* XGBoost built-in framework: https://gitlab.com/juliensimon/dlnotebooks/-/blob/master/sagemaker/09-XGBoost-script-mode.ipynb
* BYO with Scikit-learn: https://github.com/awslabs/amazon-sagemaker-examples/blob/master/advanced_functionality/scikit_bring_your_own/scikit_bring_your_own.ipynb
* Deploying XGBoost with mlflow: https://youtu.be/jpZSp9O8_ew
* New model format: https://xgboost.readthedocs.io/en/latest/tutorials/saving_model.html
* Converting pickled models: https://github.com/dmlc/xgboost/blob/master/doc/python/convert_090to100.py

This podcast is also available in video at https://youtu.be/w0F4z0dMdzI.

For more content, follow me on:
* Medium https://medium.com/@julsimon
* Twitter https://twitter.com/@julsimon

speaker 0:   0:00
Hi, this is Julian. Formidable. Yes. Welcome to Episode 11 Off My Podcast This episode is an ex J boost special A CZ. You probably know X J. Bruce 10 came out just a few days ago, and I thought this would be a good opportunity to show you the different ways in which you can train and deploy Ex Tables models on stage maker. So don't forget to subscribe to my channel for future videos on Let's Dive into Extra Boost. Let's take a look at the new features in extra boost 10 The 1st 1 is better performance scaling on multi core CPI use and they claim up to five x speed up on Intel CP use with many course. So that's pretty good, because on AWS, you can definitely use Intel CPS With many course, you have quite a choice off training instances. So whether you run extra booze on easy to directly or whether you run it on the manage service like sage maker, that should make a difference. So surely something worth benchmarking pretty soon. Well, the next one is ah, is good news for me and everybody using Mac OS. It's now easier to install extra boost on Mac OS, and that's not a tiny thing. And there's also distributed extra boost on Cooper Nature's. If that's your thing, they have a tutorial for it. So probably you want to check that out as well. It will be interesting to try and run this on the E K s and see what happens. Maybe I can add this on my to do list. Ruby bindings for extra boost For Ruby developers Ah, desk interface of desk is a very, very cool framework for distributed computing, and ah, and so now. Actually, it was can natively use it with the multiple GP use, So that's pretty interesting. Desk is one of those things that you keep hearing more and more about so again, something for the to do list. First class support for a coup de f did of frames and coop I arrays. So again Maur integration off extra boost with and video GP use, which seems to be a very, very strong theme in extra boost 10 And that's kind of interesting because most customers I meet actually run except with some CPU, and they're quite happy with it. But I guess as they train on larger and larger data sets, the appeal of GP use becomes more and more important. So it's good to see that extra boost is supporting on integrating more with GP use again. Something worth testing. What else do we have? A ranking on GPU External memory for GPU training. So that's interesting again, if you have. If your data set is bigger, that then what's your GP? You can accommodate an extra boost. Can can manage that. It used to be able to do that on CPU. Could actually train on the on larger than memory. UH, Dennis, it's that looks like it can do that on GPS as well. Improvement to the secular interface. Good news as well. Um, lots of stuff on extra boost. Forge a spark if that's your thing. And I guess the last one I want to mention is on, and I'll talk about that again. When we look at the examples, they're experimenting with a new format to save models. So I guess you know, most of us have bean using pickle thio to serialize extra boost models in python. But this creator a bunch of problems accurately and and they're moving from from this format. Thio more open and more portable, I guess, Format, which is Jason based. And they have new AP ice for that again, I'll point you. I'll point you thio the, um But it looks like if you have pickle models, you will have to retrain them and save them using the new format. Um, it looks like loading pickle models in extra boost 10 and later is something that can be a problem. So it looks like we have to retrain. Okay. And then ah, more things. Buck fixes et cetera, et cetera. Okay, so this is really, really a major release, right? Lots of GPU improvements on dhe, um, lots of scaling improvements as well. So that's pretty good. And, yeah, there's already a one. A one. Patrick, he's looks like they There was a tiny thing that made it past validation. Anyone? I was 011 of Oh, okay. Anyway. Okay. So now let's look at how we can run extra boosts on Ah, sage maker. First way to run X J boost on onstage maker is to use it as a built in argo. And this was the original way. As you know, sage Maker as a collection of built in Argos, extra boost is one of them. And what this means is you have, ah, built in extra boost container that you can use directly. Just set hyper parameters, defined the location of data and train. Okay, so I guess you know, you've seen this before. Um, but really quickly. This is how you would do it. Ah, and basically, you would just grab the container right this line. Here, grab the container for the extra boost I'll go in your region and then use the sagemaker estimator, which is the generator object to configure training jobs, pass it the name of that container and then set hyper parameters for for extra boost. And again, the list of hyper parameters is pretty much the exact same list that you would see injected with documentation. Okay, this isn't different. Um, and then you would just train. Okay, So very simple. Select the right container said hyper parameters and train. So that's the That's the first way. Um, you have to pay attention to a couple of things. First, you can actually select the ex Tables version that you want to use right and you can see here in the dark. So when you grab the name of the image of the container image, there's a paramedical repo version where you can select the actual version that you want. Okay, so I would recommend that we're using this 1.90 dash to That's the latest version at the time of recording. I'm hoping very soon we'll have 10 here. And you definitely want this one because it is the 1st 1 to support Sage McCurdy Burger, that nice service that lets you could figure Deborah debugging rules and inspect your training jobs as they go. So if you use anything older than this, save me, Cody bugger won't be available. And generally, you know you want the later version. Right now, the second way you can use extra boost on sage maker is to use it as a built in framework similar to chance a flow pytorch, mxnet, et cetera, et cetera. And just like for those frameworks, the container that we use here is open sourced and you can find it on. Get up. I will put all those links in the video description as usual and you can grab this container. You can build it on your local machine. You can run it on your local machine. You can customize it et cetera, et cetera. Okay, so the way this works is extremely similar to ah, what we do with the other frameworks. So here we're using the X J boost object from the U. K passing script defining infrastructure, choir moons, setting hamper parameters. The Freeman version as well, et cetera, et cetera. Okay, so this is exactly the same, um, the same way as all the frameworks. Alright, here's ah simple extra boost. Example where I am training a classifier on my usual day to set the direct marketing day too said that I keep using everywhere and, um, I am using script mode. Okay, so if you're not familiar with script mode, script mode is the way to interface existing framework code with stage maker. And this is really, really simple in it's all about receiving hyper parameters from the command line receiving environment, variables that defined the location of the training script, up receiving environment, variables that define the location of the training, said the validation set and where to save the model, and that's about it. So it is very simple. If you have existing extra boost code, it's not a big deal to convert it to script mode. Okay, so what am I doing here? Well, importing extra boost, obviously, arguments from the command line. So here. I just passed a single hyper parameter max depth. But you can add all the hyper parameters that you want. Okay? The only thing is, they need to be commanding arguments. And then I grab those three environment variables that I mentioned. Okay, so training, sad validation. Said location. Uh, where to save the model. Okay, so grab those. And then using the environment variables, I load the data set. So it's already split in training and valuation. So I already have those two channels and loading my dates. That really means reading CIA's V file with pandas. The labels are just that Yes. Yes. Column. And the samples are the everything but the Yes, yes, column. Okay. I'm sure you've seen these dates that before. I've used it in so many videos before. Okay, So loading the data set and then creating my extra was classy fire. So I am building a classy fires. A binary logistic is my objective. I'm using the every under curve metrics metric because it's ah, it's a good one for this data set. And I'm passing that Max depth hyper parameter again. I could set more, and I just want to keep it simple here. Then I train score on the valuation, said I pray in the area under a curve, and I saved the model. Okay. And you'll notice I am not pickling the model here. Okay, I am using that saved model A p I that saves in Ah, uh, in the binary format. Okay, that new Jason format is described at this page here. Okay, We're going to take a look at that. Okay, So how do I use this? Ah, script. Well, I have a vanilla extract script mode notebook here, and, uh, what I'm doing is downloading the data set. Ah, I am one out encoding. Like I said, dropping the yes. No column. I just need one label not to, and ah, I'm splitting. The date said between training and validation, saving those two fragments as he has the files defining the location. Where to save the model And then, just like we've seen, I use this extra boost estimator passing my script. I can train locally for quick debugging. Ah, I said the framework version gay ladies version available for now and then I can see my model training. I see the way you see that I printed out. So once the model has been trained, eye can deploy it, just using the deploy a p I. And after a few minutes, I can I can predict with it. Okay, so here I'm extracting one sample from the validation set, and that's the data frame. Okay, so I need to convert it two c s V. I can print that, uh, sample and I can use the invoke and points a p i from photo three to send the payload as C s V to the model and then read the answer. Which is that, of course, the probability between zero and one for that sample. Okay, so I I like to use the boat of three FBI because most of the time, um, you will invoke models that have been already deployed maybe days ago, so Ah, so you don't have access to the two that estimator object and you can truly call extra boost X j b estimator dot Predict on it. You can do that as well. You could recreate the object and Cole predict. But generally I find this to be easier. Okay, so So that's how you do it here. Notice that I used C s V. Um, I think you could send ah c s v lieb a ZPM, and ah, and probably protoblood off. If you want to use something else, then you need to write Ah, function that will an input FN function that will convert the input from the former that you send thio to the format that extra boost me. So, for example, if you want to send them by, um, no. Um pie, uh, raise directly here. That's possible. But you need to write that input FN function. And I I didn't do that. Okay? One thing you have to write is the model FN function. Okay, this is not provided by the container, so you need to do that. But that's pretty basic. In 99% of situations, just load the model that you saved, right? Remember, we saved the exit. Was model here using the same model a p I. So I'm just using the load model e p i to to load it again in model F n. Okay, so remember this one is mandatory. Okay? It will complain if you if you forget it anyway. Okay. Um, while we're on the topical off loading and saving models, this is the bit I wanted to cover quickly. So this is the new Jason based format for extra boost models. There's quite a bit to read on, and it's quite interesting. You know why they wanna switch to this representation for models? What it means etcetera, etcetera. So quite a bit of ah, of stuff to read. And what? The one thing that caught my eye is this s O if you want to load pickle models. So typically, if you have models coming from a J boost 00.90 which is what we have on stage maker at the moment, Um, you can try loading the models within you, uh, with the new version of extra boost. But they're kind of clear that, you know, it could be problematic. So in the x ray was repo, they actually provide a script to try and convert pickle models to the new format. Ah, but they advise you not to use it when stability is needed. So give it a shot. But again, I think retraining is probably a better way. Okay, so I'll put those links as well in the description. And just to closed this extra special. Well, of course, you could use a completely custom container. Ah, and, ah, last week, I showed you how to do this with ml flow. Um, take a look at this. If you haven't, it's I think it's pretty cool. I train a local model on my Mac and then used ml flow to deploy, choosing a custom container on stage maker. And the good thing is, M l feel actually built a container for you. So it's a completely hands off operation here. Or you could build your own sagemaker container because you want to be sure what's in it, and you have specific requirements and what not And, ah, well, we have a bunch of examples off that, and ah, and one way would be maybe to try and and do this with psychic learn as well. Um, so bring your own container is definitely an option if you like writing doctor files and and and all that kind of stuff. Not my cup of tea, personally. But the option is here and again, our reference, some some examples that show you how to do that. Okay, so these are really the 33 options. Go with a built in our logo. If you don't want to write machine learning code, you just want to use off the shelf containers and just said hyper parameters. That's nice. Um, the downside is, of course, if you need to do extra processing before or after the actual training, um, or prediction, then you will have to do that stuff outside of that container. Okay, Um and I guess in this case, the second option is interesting. Using the framework mode where you can bring your own code, your own extra boost script. Just like I showed you. You could have ah, processing steps in their post processing steps in there. So you have more control over the code. And if you really want to build everything yourself, then, uh, you know, bringing your own container is definitely possible. All right. I think that's Ah what I wanted to tell you about extra boost. Okay, that's it for Episode 11. I hope you learned a few things. Thank you for watching. Don't forget to subscribe to my channel to be notified of future videos and until next time, keep rocking.