DOWNLOAD SPARK PACKAGES INSTALL

When working with a small number of packages this is an easy approach to get the exact version setup so you may prefer it even if the library is available in a repository.

This method is required if the code is not available in a repository that your Spark pool can reach.

DOWNLOAD SPARK PACKAGES INSTALL

Installing WHL files (for Python packages)ДЄnother option to install Python libraries is to add individual WHL files that will be installed when the Spark pool starts. These will only be available for this session so itвЂ™s best suited for testing out new libraries.ДЇor that you go to the Settings from your PySpark notebook.ДЇor session packages you need to use a YML file to define the libraries and related settings. Less common is to add the libraries only to your current session. The package will be installed the next time a sessions starts up. Upload the file from the Requirements files section. In Azure Synapse Studio, browse to a specific Apache Spark pool and select Packages. According to the documentation you can use both requirements.txt and environment.yml together.

Many Python developers have used the requirements.txt format, but if you want to specify a specific channel to get libraries from the environment.yml file is needed. I recommend using whichever file type you have the most experience using. This can be a requirements.txt file or an environment.yml file. In this case you can add a file that specifies the package names and versions to install. It is fairly simple when the required libraries exist in a repository like Conda or PyPI (Python Packaging Index). Installing Python packages from PyPI, Conda, etcДЄzure Synapse Spark pools come with all the Anaconda libraries installed, but you will likely find a need for additional libraries or different versions of libraries.

Download spark packages